deep_learning/3_Deep_learning_y_NLP.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "@webio": {
      "lastCommId": null,
      "lastKernelId": null
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.11"
    },
    "colab": {
      "name": "3_Deep_learning_y_NLP.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hrd4RZ9BTgSZ"
      },
      "source": [
        "## Redes neuronales para análisis de sentimiento sobre IMBD\n",
        "\n",
        "En este cuaderno entrenaremos y evaluaremos un modelo sencillo basado en la arquitectura Transforer para clasificar reviews de películas en positivas o negativas. Para hacerlo, además de utilizar pytorch, utilizaremos el paquete adicional `torchtext` , que nos ofrece algunas funciones útiles para trabajar con texto."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0jG2k7aATgSd"
      },
      "source": [
        "#conda install -c pytorch torchtext"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zPVzbzxJTgSe"
      },
      "source": [
        "import torch\n",
        "from torchtext.legacy import data\n",
        "from torchtext.legacy import datasets\n",
        "import random\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim"
      ],
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NUehR0j2TgSf"
      },
      "source": [
        "## Descargando y procesando los datos"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-qX2wsBJTgSg"
      },
      "source": [
        "En primer lugar debemos tener instalado `spacy`, con el tokenizador ingés estándar (`python -m spacy download en`), para separar las reviews en palabras.\n",
        "\n",
        "Ahora, definiremos dos variables, una el `TEXT`, que será nuestra $x$, y la otra `LABEL`, que será nuestra $y$. Fijaos en que en el caso de las features tenemos que especificar el tokenizador que vamos a utilizar, mientras que para la label simplemente será un valor numérico"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "74uT8x1qTgSg"
      },
      "source": [
        "TEXT = data.Field(tokenize = 'spacy')\n",
        "LABEL = data.LabelField(dtype = torch.float)"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bfi_pg4vTgSh"
      },
      "source": [
        "Ahora, dividiremos los datos en un conjunto de entrenamiento y otro de evaluación. Fijémonos en que, al igual que como en el MNIST, torchtext ofrece un submódulo datasets que contiene los datasets de NLP más populares, como el IMDB."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7MfCQufVTgSh",
        "outputId": "a79559e2-7ec6-4332-f695-09d6f8b9be6b"
      },
      "source": [
        "train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "downloading aclImdb_v1.tar.gz\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "aclImdb_v1.tar.gz: 100%|██████████| 84.1M/84.1M [00:05<00:00, 15.6MB/s]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M8PHgePOTgSi"
      },
      "source": [
        "Echemos un vistazo al primer ejemplo de entrenamiento"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aaPTyXx4TgSi",
        "outputId": "3a7934c2-3c36-4980-fa61-a6dc4c1b27fe"
      },
      "source": [
        "print(vars(train_data.examples[-1]))"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'text': ['I', 'rarely', 'comment', 'on', 'films', 'but', 'I', \"'ve\", 'read', 'the', 'other', 'comments', 'and', 'I', 'can', 'not', 'believe', 'that', 'there', 'are', 'people', 'applauding', 'this', 'celluloid', 'rubbish', '.', 'I', 'know', 'there', 'are', 'certain', 'people', 'who', 'have', 'their', 'own', 'agenda', 'but', 'lets', 'take', 'it', 'on', 'merit', ';', 'poorly', 'acted', ',', 'badly', 'shot', 'and', 'the', 'story', 'felt', 'as', 'the', 'director', 'was', 'making', 'it', 'up', 'as', 'he', 'was', 'going', 'along', '.', 'I', 'am', 'not', 'going', 'to', 'focus', 'on', 'the', 'sexual', 'aspect', 'of', 'the', 'film', 'involving', 'little', 'kids', 'as', 'the', 'makers', 'of', 'the', 'film', 'obviously', 'knew', 'what', 'they', 'wanted', 'and', 'what', 'their', 'audience', 'would', 'want', '.', 'All', 'I', 'can', 'say', 'is', 'it', 'is', 'a', 'terrible', 'film', ',', 'the', 'content', 'is', 'poor', 'and', 'offensive', ',', 'the', 'production', 'is', 'amateurish', 'and', 'I', 'am', 'glad', 'they', 'could', 'not', 'make', 'a', 'film', 'like', 'this', 'legally', 'today'], 'label': 'neg'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NIedNyGaTgSi"
      },
      "source": [
        "Como esperamos, la review ha sido clasificada como positiva `pos`.\n",
        "\n",
        "Ahora, haremos otra partición para validación"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cJ-BAPLpTgSj"
      },
      "source": [
        "SEED = 1230245\n",
        "train_data, valid_data = train_data.split(random_state = random.seed(SEED))"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HWt4cHThTgSj"
      },
      "source": [
        "**Ejercicio** Para el conjunto de entrenamiento, y cada clase, ¿cuánto mide la review más larga y la más corta (en tokens)?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QTssDr7dTgSk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bd4693fd-0665-4b29-92b7-bb92368a57b9"
      },
      "source": [
        "max([len(vars(example)['text']) for example in train_data.examples if vars(example)['label'] == 'pos'])"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2789"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sSt98xI8jRUQ",
        "outputId": "f34aa7d5-9cab-4317-afe0-58afc956992a"
      },
      "source": [
        "min([len(vars(example)['text']) for example in train_data.examples if vars(example)['label'] == 'pos'])"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "20"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NKNjdUZ7jRMc",
        "outputId": "5e7cc5c3-73a9-4c77-dd17-48a3ff5137e5"
      },
      "source": [
        "max([len(vars(example)['text']) for example in train_data.examples if vars(example)['label'] == 'neg'])"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1827"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xbStX6gajRCU",
        "outputId": "1ffaf8b1-9be9-4a63-8ef4-6c2aac2f2cd7"
      },
      "source": [
        "min([len(vars(example)['text']) for example in train_data.examples if vars(example)['label'] == 'neg'])"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "12"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i3B01NudTgSk"
      },
      "source": [
        "### Construyendo el vocabulario\n",
        "\n",
        "Ahora que ya tenemos para la $x$ una lista de palabras, ¿cómo podemos convertirla en un vector?\n",
        "\n",
        "Now we have for $x$ a list of words, but how we can convert that to a vector?\n",
        "\n",
        "El primer paso es definir un vocabulario, esto es, un subconjunto de todas las palabras que aparecen en el dataset, y solo nos fijaremos en esas palabras. Nuestro vocabulario tendrá un tamaño de 5000 palabras. Hay muchas formas de construirlo, pero la más usual es la de que quedarnos con las 5000 palabras más comunes en nuestro dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7ZsrJamFTgSk"
      },
      "source": [
        "MAX_VOCAB_SIZE = 5000\n",
        "\n",
        "TEXT.build_vocab(train_data, max_size = MAX_VOCAB_SIZE)\n",
        "LABEL.build_vocab(train_data)"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "72jt2OzLTgSk"
      },
      "source": [
        "Esto es, este vocabulario recién creado, a cada palabra $w_i$ le asigna un entero distinto. El vocabulario no es más que una aplicación $V : W \\rightarrow \\lbrace 0, \\ldots, 5002 \\rbrace \\subset \\mathbb{N}$ del conjunto de palabras $W$ a los respectivos números enteros.\n",
        "\n",
        "La inclusión de dos número enteros adicionales (por eso es 5002 y no 5000) es debida a que necesitamos un código especial para las palabras \"raras\" que no están entre las 5000 más comunes, y otro número especial para indicar el final de la frase (usarlo como \"padding\")."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kB0Y9veDTgSl"
      },
      "source": [
        "### Exploración de features\n",
        "\n",
        "Ahora, podemos acceder al atributo `TEXT.vocab` para explorar el dataset. Por ejemplo, podemos encontrar los 20 tokens más comunes:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Zxf_cgwlTgSl",
        "outputId": "c714b63c-26ef-4592-d791-a8faeabd8ae0"
      },
      "source": [
        "print(TEXT.vocab.freqs.most_common(20))"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('the', 204939), (',', 193428), ('.', 166611), ('a', 110046), ('and', 109807), ('of', 101629), ('to', 94217), ('is', 77027), ('in', 61493), ('I', 54393), ('it', 54084), ('that', 49441), ('\"', 44019), (\"'s\", 43582), ('this', 42423), ('-', 36926), ('/><br', 35892), ('was', 35099), ('as', 30466), ('with', 29954)]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N25ZEXy1TgSl"
      },
      "source": [
        "Por ejemplo, podemos ver que `the`es la palabra más común, apareciendo 203504 veces en nuestro corpus.\n",
        "\n",
        "El método stoi calcula la función $V$ anterior ('string to integer')\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "enDLxC8ATgSm",
        "outputId": "bf3ef87c-0fee-4ef5-938f-6cbef066599d"
      },
      "source": [
        "TEXT.vocab.stoi['the']"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ndz5KyoPTgSm"
      },
      "source": [
        "También tenemos acceso a la función inversa, de enteros a strings:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rvsbhJViTgSm",
        "outputId": "eeac116d-a745-4147-9116-2d7e6244c8cb"
      },
      "source": [
        "print(TEXT.vocab.itos[:10])"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['<unk>', '<pad>', 'the', ',', '.', 'a', 'and', 'of', 'to', 'is']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LChz2C2cTgSm"
      },
      "source": [
        "**Ejercicio** ¿Qué código numérico le ha asignado a la clase positiva?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cEl9Vdb1TgSm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a2c6bb0a-48cf-4a54-ce8b-77127cd25b23"
      },
      "source": [
        "LABEL.vocab.stoi['pos']"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ogRLIvQZTgSm"
      },
      "source": [
        "## Definición del modelo: capas de Transformers\n",
        "\n",
        "Ahora que ya tenemos el vocabulario, podemos representar cada palabra como un vector one-hot, sobre el espacio $\\lbrace 0, 1 \\rbrace^{5002}$. Este espacio es muy disperso (\"vacío\"), y hace que cada palabra se encuentre a la misma distancia que cualquier otra. Así que lo primero que podemos hacer en NLP profundo es aplicar una proyección lineal a un espacio de dimensionalidad mucho más baja. Esto es, para la iésima palabra, haremos $h_i = W x_i$, donde $W$ es una matriz de tamaño $100 \\times 5002$. El 100 se refiere a la dimensión del \"embedding\". Para más información, podéis consultar por *word embeddings*.\n",
        "\n",
        "Después, tras haber calculado esta nueva representación de las palabras, podemos aplicar una capa de Transformer, una arquitectura que está diseñada para captar patrones en secuencias de símbolos, con el objetivo de aprender qué partes de una frase son más informativas respecto a su sentimiento. La arquitectura Transformer es muy reciente (https://arxiv.org/abs/1706.03762) y, al contrario que las antecesoras redes recurrentes, es más fácil de paralelizar por lo que su entrenamiento es mucho más rápido.\n",
        "\n",
        "Para ver los detalles de la arquitectura Transformer podéis consultar los siguientes artículos:\n",
        "\n",
        "* https://jalammar.github.io/illustrated-transformer/\n",
        "\n",
        "* https://nlp.seas.harvard.edu/2018/04/03/attention.html"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nkPQICoGTgSn"
      },
      "source": [
        "class Transformer(nn.Module):\n",
        "    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, n_layers, \n",
        "                    dropout, pad_idx):\n",
        "        super().__init__()\n",
        "        \n",
        "        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx = pad_idx)\n",
        "        self.enc = nn.TransformerEncoderLayer(embedding_dim, 4, hidden_dim)\n",
        "        self.fc = nn.Linear(embedding_dim, output_dim)\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        \n",
        "    def forward(self, text):\n",
        "        \n",
        "        embedded = self.dropout(self.embedding(text))\n",
        "        hidden = self.enc(embedded)\n",
        "        hidden = hidden.mean(dim=0)\n",
        "        return self.fc(hidden)"
      ],
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lt8qwaagTgSo"
      },
      "source": [
        "Como en el cuaderno anterior, definimos ahora los hiperparámetros y los iteradores sobre el dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BRnIXQV8TgSo"
      },
      "source": [
        "BATCH_SIZE = 16\n",
        "\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "\n",
        "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(\n",
        "    (train_data, valid_data, test_data), \n",
        "    batch_size = BATCH_SIZE,\n",
        "    device = device)\n",
        "\n",
        "# Las dos dimensiones de la matriz de embeddings W:\n",
        "INPUT_DIM = len(TEXT.vocab)\n",
        "EMBEDDING_DIM = 100\n",
        "\n",
        "#  Como solo queremos predecir positivo o negativo, simplemente el output es un número escalar\n",
        "# en R^1 (usando la sigmoide para obtener una probabilidad)\n",
        "OUTPUT_DIM = 1\n",
        "# Apilaremos dos capas de Transformers\n",
        "N_LAYERS = 2\n",
        "# Especificamos algo de dropout para regularizar la red neuronal\n",
        "DROPOUT = 0.5\n",
        "PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]\n",
        "\n",
        "model = Transformer(INPUT_DIM, \n",
        "            16, \n",
        "            EMBEDDING_DIM // 10, \n",
        "            OUTPUT_DIM, \n",
        "            N_LAYERS, \n",
        "            DROPOUT, \n",
        "            PAD_IDX)\n",
        "\n",
        "model = model.to(device)\n",
        "\n",
        "\n",
        "optimizer = optim.Adam(model.parameters())\n",
        "criterion = nn.BCEWithLogitsLoss()\n",
        "criterion = criterion.to(device)"
      ],
      "execution_count": 27,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "67f4grLqTgSo"
      },
      "source": [
        "## Entrenando el modelo\n",
        "\n",
        "El resto del cuaderno es como el anterior, solo tenemos que definir una función auxiliar para calcular la tasa de acierto y las funciones de entrenamiento y evaluación"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HzS2UZJeTgSp"
      },
      "source": [
        "def binary_accuracy(preds, y):\n",
        "    \"\"\"\n",
        "    Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8\n",
        "    \"\"\"\n",
        "    rounded_preds = torch.round(torch.sigmoid(preds))\n",
        "    correct = (rounded_preds == y).float()\n",
        "    acc = correct.sum() / len(correct)\n",
        "    return acc"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kxxP0PThTgSp"
      },
      "source": [
        "def train(model, iterator, optimizer, criterion):\n",
        "    \n",
        "    epoch_loss = 0\n",
        "    epoch_acc = 0\n",
        "    \n",
        "    model.train()\n",
        "    \n",
        "    for (text, cls) in iterator:\n",
        "        \n",
        "        optimizer.zero_grad()\n",
        "        predictions = model(text).squeeze(1)\n",
        "        loss = criterion(predictions, cls)\n",
        "        acc = binary_accuracy(predictions, cls)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        \n",
        "        epoch_loss += loss.item()\n",
        "        epoch_acc += acc.item()\n",
        "        \n",
        "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5jIBrClETgSp"
      },
      "source": [
        "def evaluate(model, iterator, criterion):\n",
        "    \n",
        "    epoch_loss = 0\n",
        "    epoch_acc = 0\n",
        "    \n",
        "    model.eval()\n",
        "    \n",
        "    with torch.no_grad():\n",
        "    \n",
        "        for (text, cls) in iterator:\n",
        "\n",
        "            predictions = model(text).squeeze(1)\n",
        "            loss = criterion(predictions, cls)\n",
        "            acc = binary_accuracy(predictions, cls)\n",
        "\n",
        "            epoch_loss += loss.item()\n",
        "            epoch_acc += acc.item()\n",
        "        \n",
        "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
      ],
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xzKo0PORTgSp"
      },
      "source": [
        "Tras definir estos bucles, entrenamos el modelo"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ABFZW0jBYakV"
      },
      "source": [
        "Ojo: para tardar menos, hemos puesto como conjunto de train el de validación (que es más pequeño) y como validación el de test."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 579
        },
        "id": "y04s8iE6TgSq",
        "outputId": "16ce6fa9-eecb-4b1e-c1c0-f6c0f7d15004"
      },
      "source": [
        "N_EPOCHS = 5\n",
        "\n",
        "for epoch in range(N_EPOCHS):\n",
        "\n",
        "    train_loss, train_acc = train(model, valid_iterator, optimizer, criterion)\n",
        "    valid_loss, valid_acc = evaluate(model, test_iterator, criterion)\n",
        "    \n",
        "    torch.save(model.state_dict(), 'model.pt' + str(epoch))\n",
        "    \n",
        "    print(f'Epoch: {epoch+1:02}')\n",
        "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
        "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 01\n",
            "\tTrain Loss: 0.676 | Train Acc: 56.75%\n",
            "\t Val. Loss: 0.625 |  Val. Acc: 64.96%\n",
            "Epoch: 02\n",
            "\tTrain Loss: 0.605 | Train Acc: 67.00%\n",
            "\t Val. Loss: 0.544 |  Val. Acc: 72.66%\n",
            "Epoch: 03\n",
            "\tTrain Loss: 0.551 | Train Acc: 72.17%\n",
            "\t Val. Loss: 0.506 |  Val. Acc: 75.28%\n",
            "Epoch: 04\n",
            "\tTrain Loss: 0.501 | Train Acc: 75.74%\n",
            "\t Val. Loss: 0.487 |  Val. Acc: 77.06%\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-72-01d6030ada58>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mepoch\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mN_EPOCHS\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m     \u001b[0mtrain_loss\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtrain_acc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalid_iterator\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcriterion\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m     \u001b[0mvalid_loss\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalid_acc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mevaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtest_iterator\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcriterion\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-66-1231bab46945>\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(model, iterator, optimizer, criterion)\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m         \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m         \u001b[0mpredictions\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqueeze\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m         \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcriterion\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpredictions\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcls\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m         \u001b[0macc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbinary_accuracy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpredictions\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcls\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1049\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1050\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1051\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1052\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1053\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-63-64215bd85dce>\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, text)\u001b[0m\n\u001b[1;32m     12\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m         \u001b[0membedded\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdropout\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0membedding\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 14\u001b[0;31m         \u001b[0mhidden\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0membedded\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     15\u001b[0m         \u001b[0mhidden\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhidden\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdim\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhidden\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1049\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1050\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1051\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1052\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1053\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/transformer.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, src, src_mask, src_key_padding_mask)\u001b[0m\n\u001b[1;32m    319\u001b[0m         \"\"\"\n\u001b[1;32m    320\u001b[0m         src2 = self.self_attn(src, src, src, attn_mask=src_mask,\n\u001b[0;32m--> 321\u001b[0;31m                               key_padding_mask=src_key_padding_mask)[0]\n\u001b[0m\u001b[1;32m    322\u001b[0m         \u001b[0msrc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msrc\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdropout1\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msrc2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    323\u001b[0m         \u001b[0msrc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnorm1\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msrc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1049\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1050\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1051\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1052\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1053\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/activation.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, query, key, value, key_padding_mask, need_weights, attn_mask)\u001b[0m\n\u001b[1;32m   1036\u001b[0m                 \u001b[0mtraining\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtraining\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1037\u001b[0m                 \u001b[0mkey_padding_mask\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mkey_padding_mask\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mneed_weights\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mneed_weights\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1038\u001b[0;31m                 attn_mask=attn_mask)\n\u001b[0m\u001b[1;32m   1039\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbatch_first\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1040\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mattn_output\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtranspose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mattn_output_weights\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py\u001b[0m in \u001b[0;36mmulti_head_attention_forward\u001b[0;34m(query, key, value, embed_dim_to_check, num_heads, in_proj_weight, in_proj_bias, bias_k, bias_v, add_zero_attn, dropout_p, out_proj_weight, out_proj_bias, training, key_padding_mask, need_weights, attn_mask, use_separate_proj_weight, q_proj_weight, k_proj_weight, v_proj_weight, static_k, static_v)\u001b[0m\n\u001b[1;32m   5080\u001b[0m     \u001b[0;31m# (deep breath) calculate attention and out projection\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   5081\u001b[0m     \u001b[0;31m#\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 5082\u001b[0;31m     \u001b[0mattn_output\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mattn_output_weights\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_scaled_dot_product_attention\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mq\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mattn_mask\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdropout_p\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   5083\u001b[0m     \u001b[0mattn_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mattn_output\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtranspose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontiguous\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mview\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtgt_len\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbsz\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0membed_dim\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   5084\u001b[0m     \u001b[0mattn_output\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlinear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mattn_output\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mout_proj_weight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mout_proj_bias\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py\u001b[0m in \u001b[0;36m_scaled_dot_product_attention\u001b[0;34m(q, k, v, attn_mask, dropout_p)\u001b[0m\n\u001b[1;32m   4823\u001b[0m     \u001b[0mq\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mq\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mmath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqrt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mE\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4824\u001b[0m     \u001b[0;31m# (B, Nt, E) x (B, E, Ns) -> (B, Nt, Ns)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 4825\u001b[0;31m     \u001b[0mattn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbmm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mq\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtranspose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   4826\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mattn_mask\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4827\u001b[0m         \u001b[0mattn\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mattn_mask\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "49_oZvxsfGpu"
      },
      "source": [
        "Como vemos, aun estando en GPU es lento. Si tenéis un rato, podéis dejarlo unas cuantas épocas más, y la accuracy debería llegar al 90% aproximadamente. \n",
        "\n",
        "En la clase, vamos a probar con un modelo más sencillo, en el ejercicio del final"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jH4VDqvCTgSq"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tIxvH95NTgSq"
      },
      "source": [
        "**Ejercicio** ¿Podemos hacerlo mejor? Trata de modificar los hiperparámetros para ver como cambia la tasa de acierto"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XkRgZUEdTgSr"
      },
      "source": [
        "**Ejercicio** Sustituye la capa de Transformer por una lineal y observa cómo cambia la tasa de acierto.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yAE8KeGHb6f9"
      },
      "source": [
        "class MLP_NLP(nn.Module):\n",
        "    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, n_layers, \n",
        "                    dropout, pad_idx):\n",
        "        super().__init__()\n",
        "        \n",
        "        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx = pad_idx)\n",
        "        self.enc = nn.Linear(embedding_dim, hidden_dim)\n",
        "        self.fc = nn.Linear(hidden_dim, output_dim)\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        \n",
        "    def forward(self, text):\n",
        "        \n",
        "        embedded = self.dropout(self.embedding(text))\n",
        "        hidden = F.relu(self.enc(embedded))\n",
        "        hidden = hidden.mean(dim=0)\n",
        "        return self.fc(hidden)"
      ],
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7BfI3ByEj5mk"
      },
      "source": [
        "model = MLP_NLP(INPUT_DIM, \n",
        "            16, \n",
        "            EMBEDDING_DIM // 10, \n",
        "            OUTPUT_DIM, \n",
        "            N_LAYERS, \n",
        "            DROPOUT, \n",
        "            PAD_IDX)\n",
        "\n",
        "model = model.to(device)\n",
        "\n",
        "optimizer = optim.Adam(model.parameters())\n",
        "criterion = nn.BCEWithLogitsLoss()\n",
        "criterion = criterion.to(device)"
      ],
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "84gGIweVjnnu",
        "outputId": "2ffe801c-9ab2-4b04-abc6-cbe2acdc0f46"
      },
      "source": [
        "N_EPOCHS = 5\n",
        "\n",
        "for epoch in range(N_EPOCHS):\n",
        "\n",
        "    train_loss, train_acc = train(model, valid_iterator, optimizer, criterion)\n",
        "    valid_loss, valid_acc = evaluate(model, test_iterator, criterion)\n",
        "    \n",
        "    torch.save(model.state_dict(), 'model.pt' + str(epoch))\n",
        "    \n",
        "    print(f'Epoch: {epoch+1:02}')\n",
        "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
        "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 01\n",
            "\tTrain Loss: 0.692 | Train Acc: 51.62%\n",
            "\t Val. Loss: 0.690 |  Val. Acc: 50.04%\n",
            "Epoch: 02\n",
            "\tTrain Loss: 0.682 | Train Acc: 59.03%\n",
            "\t Val. Loss: 0.674 |  Val. Acc: 55.91%\n",
            "Epoch: 03\n",
            "\tTrain Loss: 0.644 | Train Acc: 68.45%\n",
            "\t Val. Loss: 0.614 |  Val. Acc: 69.03%\n",
            "Epoch: 04\n",
            "\tTrain Loss: 0.560 | Train Acc: 74.57%\n",
            "\t Val. Loss: 0.531 |  Val. Acc: 74.57%\n",
            "Epoch: 05\n",
            "\tTrain Loss: 0.488 | Train Acc: 78.41%\n",
            "\t Val. Loss: 0.479 |  Val. Acc: 77.22%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vASoqAw_kAEx"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}