Notebooks/03_AITEXTIFICATION_Modelos.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "NyGZjrod_7ro",
        "p4_3A1YDyjf9",
        "-zkIZLo-hg3a",
        "LYf9P4w76y1S",
        "-LQvpRc8yIgc",
        "R8h2nmsIH9By"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# 📚 PROYECTO AI-TEXTIFICATION\n",
        "## 👅 Procesamiento de Lenguaje Natural\n",
        "## 💻 Instituto de Investigaciones en Matemáticas Aplicadas y en Sistemas\n",
        "## 🏫 Universidad Nacional Autónoma de México\n",
        "\n",
        "<hr>\n",
        "\n",
        "### 🤖 AI-TEXTIFICATION\n",
        "### 📓 CUADERNO [03]: MODELOS\n",
        "### 📄 Detección de autoría en textos AI - Humanos:\n",
        "\n",
        "#### 🔵 **Tarea A**: Clasificación Binaria:\n",
        "1. Texto de Humano.\n",
        "2. Texto de Inteligencia Artificial.\n",
        "\n",
        "#### 🔵 **Tarea B**: Clasificación Multiclase:\n",
        "1. Texto de ChatGPT.\n",
        "2. Texto de Cohere.\n",
        "3. Texto de Davinci.\n",
        "4. Texto de Dolly.\n",
        "5. Texto de Humano.\n",
        "\n",
        "**👬 Autores:**\n",
        "* León Rosas Manuel Alejandro.\n",
        "* Ramos Herrera Iván Alejandro.\n"
      ],
      "metadata": {
        "id": "P7F5VNFJMckc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# [01] 🎯 Objetivo\n",
        "\n",
        "**EN ESTE DESARROLLO DE CÓDIGO SE UTILIZARÁN LOS EMBEDDINGS Y DISTINTOS DATASETS GENERADOS EN NOTEBOOKS ANTERIORES Y SE IMPLEMENTARÁN MODELOS DE CLASIFICACIÓN PARA RESOLVER (O NO) DEFINITIVAMENTE LAA TAREAS A Y B.**\n",
        "\n",
        "**Modelos propuestos:**\n",
        "\n",
        "1. Transformer(Sólo Encoder) con Auto-Atención.\n",
        "2. Red Neuronal Artificial Recurrente.\n",
        "\n",
        "**Para los datasets:**\n",
        "\n",
        "A. Original => Cleaned => Lemma => UNK => Gensim Embeddings.\n",
        "\n",
        "B. Original => Cleaned => Lemma => UNK => Own Embeddings.\n",
        "\n",
        "C. Original => Cleaned => UNK => Gensim Embeddings.\n",
        "\n",
        "D. Original => Cleaned => UNK => Own Embeddings."
      ],
      "metadata": {
        "id": "-TQwHu8N3nzR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# [02] 📦 Data & Embeddings"
      ],
      "metadata": {
        "id": "8FdZ8aThR437"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Datasets:\n",
        "TaskA_datasets = {\n",
        "  \"A\": {\n",
        "      \"train_dataset\": \"/content/drive/MyDrive/Datasets/AITextification/TaskA-TrainAB.csv\",\n",
        "      \"test_dataset\": \"/content/drive/MyDrive/Datasets/AITextification/TaskA-DevAB.csv\",\n",
        "      \"embeddings\": {\n",
        "          \"own\": False,\n",
        "          \"path\": \"/content/drive/MyDrive/Datasets/AITextification/A_gensim_model.bin\"\n",
        "      }\n",
        "  },\n",
        "  \"B\": {\n",
        "      \"train_dataset\": \"/content/drive/MyDrive/Datasets/AITextification/TaskA-TrainAB.csv\",\n",
        "      \"test_dataset\": \"/content/drive/MyDrive/Datasets/AITextification/TaskA-DevAB.csv\",\n",
        "      \"embeddings\": {\n",
        "          \"own\": True,\n",
        "          \"model\": \"/content/drive/MyDrive/Datasets/AITextification/B_embeds_model.pkl\",\n",
        "          \"word2Index\": \"/content/drive/MyDrive/Datasets/AITextification/B_word2Index.pkl\",\n",
        "          \"dataset\": \"/content/drive/MyDrive/Datasets/AITextification/B_embeds.csv\"\n",
        "      }\n",
        "  },\n",
        "  \"C\": {\n",
        "      \"train_dataset\": \"/content/drive/MyDrive/Datasets/AITextification/TaskA-TrainCD.csv\",\n",
        "      \"test_dataset\": \"/content/drive/MyDrive/Datasets/AITextification/TaskA-DevCD.csv\",\n",
        "      \"embeddings\": {\n",
        "          \"own\": False,\n",
        "          \"path\": \"/content/drive/MyDrive/Datasets/AITextification/C_gensim_model.bin\"\n",
        "      }\n",
        "  },\n",
        "  \"D\": {\n",
        "      \"train_dataset\": \"/content/drive/MyDrive/Datasets/AITextification/TaskA-TrainCD.csv\",\n",
        "      \"test_dataset\": \"/content/drive/MyDrive/Datasets/AITextification/TaskA-DevCD.csv\",\n",
        "      \"embeddings\": {\n",
        "          \"own\": True,\n",
        "          \"model\": \"/content/drive/MyDrive/Datasets/AITextification/D_embeds_model.pkl\",\n",
        "          \"word2Index\": \"/content/drive/MyDrive/Datasets/AITextification/D_word2Index.pkl\",\n",
        "          \"dataset\": \"/content/drive/MyDrive/Datasets/AITextification/D_embeds.csv\"\n",
        "      }\n",
        "  }\n",
        "}"
      ],
      "metadata": {
        "id": "u6_uCXOgYNHp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "TaskADatasetTEST = {\n",
        "    \"A\": {\n",
        "      \"train_dataset\": \"/content/drive/MyDrive/Datasets/AITextification/TaskA-TrainAB.csv\",\n",
        "      \"test_dataset\": \"/content/drive/MyDrive/Datasets/AITextification/TaskA-DevAB.csv\",\n",
        "      \"embeddings\": {\n",
        "          \"own\": False,\n",
        "          \"path\": \"/content/drive/MyDrive/Datasets/AITextification/A_gensim_model.bin\"\n",
        "      }\n",
        "  },\n",
        "  \"B\": {\n",
        "      \"train_dataset\": \"/content/drive/MyDrive/Datasets/AITextification/TaskA-TrainAB.csv\",\n",
        "      \"test_dataset\": \"/content/drive/MyDrive/Datasets/AITextification/TaskA-DevAB.csv\",\n",
        "      \"embeddings\": {\n",
        "          \"own\": True,\n",
        "          \"model\": \"/content/drive/MyDrive/Datasets/AITextification/B_embeds_model.pkl\",\n",
        "          \"word2Index\": \"/content/drive/MyDrive/Datasets/AITextification/B_word2Index.pkl\",\n",
        "          \"dataset\": \"/content/drive/MyDrive/Datasets/AITextification/B_embeds.csv\"\n",
        "      }\n",
        "  }\n",
        "}"
      ],
      "metadata": {
        "id": "isLMkwTc9y6p"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# PARA TOMAR UN DATASET Y CONVERTIRLO EN DATOS LEÍBLES POR UN MODELO:\n",
        "def get_gensim(dataset, embeddings):\n",
        "  # Obtiene el dataset:\n",
        "  dataset = pd.read_csv(dataset)\n",
        "\n",
        "  # Etiquetas \"y\":\n",
        "  y = dataset[\"label\"].values\n",
        "\n",
        "  # Obtiene las palabras únicas en el dataset:\n",
        "  unique_words = set(\" \".join(dataset[\"text\"]).split())\n",
        "\n",
        "  # Crea un diccionario de embeddings para las palabras únicas:\n",
        "  word_embeddings = {word: embeddings.wv[word] if word in embeddings.wv else np.zeros(embeddings.vector_size)\n",
        "                      for word in unique_words}\n",
        "\n",
        "  # Crear la matriz de características \"X\":\n",
        "  X = np.array([[word_embeddings[word] for word in sentence.split() if word in word_embeddings]\n",
        "                for sentence in dataset[\"text\"]])\n",
        "\n",
        "  return X, y, len(unique_words)"
      ],
      "metadata": {
        "id": "h4Bne8OjsfyS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def load_embeddings(embeds_model_path, word2Index_path, embeds_csv_path):\n",
        "  # Cargar embeddings model desde archivo pickle:\n",
        "  with open(embeds_model_path, \"rb\") as model_file:\n",
        "    embeds_model = pickle.load(model_file)\n",
        "\n",
        "  # Cargar word2Index desde archivo pickle:\n",
        "  with open(word2Index_path, \"rb\") as word2Index_file:\n",
        "    word2Index = pickle.load(word2Index_file)\n",
        "\n",
        "  # Cargar embeddings desde archivo CSV:\n",
        "  embeds_df = pd.read_csv(embeds_csv_path)\n",
        "  embeds = embeds_df.values\n",
        "\n",
        "  return embeds, word2Index, embeds_model\n",
        "\n",
        "\n",
        "def get_own(dataset, embeddings):\n",
        "  # Obtiene el dataset:\n",
        "  dataset = pd.read_csv(dataset)\n",
        "\n",
        "  # Obtiene las etiquetas \"y\":\n",
        "  y = dataset[\"label\"].values\n",
        "\n",
        "  # Obtiene las palabras únicas en el dataset:\n",
        "  unique_words = set(\" \".join(dataset[\"text\"]).split())\n",
        "\n",
        "  # Crea un diccionario de embeddings para las palabras únicas:\n",
        "  word_embeddings = {word: embeddings.get_embedding_of(word, embeddings.word2Index) if word in embeddings.word2Index else np.zeros(embeddings.embedding_size)\n",
        "                      for word in unique_words}\n",
        "\n",
        "  # Crear la matriz de características \"X\":\n",
        "  X = np.array([[word_embeddings[word] for word in sentence.split() if word in word_embeddings]\n",
        "                for sentence in dataset[\"text\"]])\n",
        "\n",
        "  return X, y, len(unique_words)"
      ],
      "metadata": {
        "id": "IpxYWSXvVBfE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.utils import shuffle\n",
        "\n",
        "# Define la función para cargar embeddings y obtener datos de entrenamiento:\n",
        "def get_data_and_embeddings(dataset_info, size=1):\n",
        "  if dataset_info[\"embeddings\"][\"own\"]:\n",
        "      embeds_model, word2Index, embeds_csv = load_embeddings(\n",
        "          dataset_info[\"embeddings\"][\"model\"],\n",
        "          dataset_info[\"embeddings\"][\"word2Index\"],\n",
        "          dataset_info[\"embeddings\"][\"dataset\"]\n",
        "      )\n",
        "      embeddings, word2Index, embeds_model = load_embeddings(embeds_model, word2Index, embeds_csv)\n",
        "      X, y, unique_words = get_own(dataset_info[\"train_dataset\"], embeds_model)\n",
        "  else:\n",
        "    embeddings = Word2Vec.load(str(dataset_info[\"embeddings\"][\"path\"]))\n",
        "    X, y, unique_words = get_gensim(dataset_info[\"train_dataset\"], embeddings)\n",
        "\n",
        "\n",
        "  # Combina X e y antes de la mezcla aleatoria:\n",
        "  combined = list(zip(X, y))\n",
        "\n",
        "  # Mezcla aleatoria:\n",
        "  combined = shuffle(combined, random_state=11)\n",
        "\n",
        "  # Desempaqueta X e y después de la mezcla aleatoria:\n",
        "  X, y = zip(*combined)\n",
        "\n",
        "  # Devuelve X, y y unique_words por separado:\n",
        "  return X[:int(len(X)*size)], y[:int(len(y)*size)], unique_words"
      ],
      "metadata": {
        "id": "d-c9JXU4rqV8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "from sklearn.utils import shuffle\n",
        "\n",
        "# Define la función para cargar embeddings y obtener datos de entrenamiento:\n",
        "def get_data_and_words(dataset_path, size=1):\n",
        "\n",
        "  # Obtiene un dataset con \"text\" y \"labels\":\n",
        "  dataset = pd.read_csv(dataset_path)\n",
        "\n",
        "  # Mezcla y obtiene una parte:\n",
        "  dataset = dataset.sample(frac=size, random_state=11)\n",
        "\n",
        "  # Separa X, y:\n",
        "  X = dataset[[\"text\"]].values.tolist()\n",
        "  y = dataset[[\"label\"]].values.tolist()\n",
        "\n",
        "  # Obtén todas las palabras únicas:\n",
        "  uniques_set = set()\n",
        "  for texto in dataset[\"text\"]:\n",
        "    palabras = texto.split()\n",
        "    uniques_set.update(palabras)\n",
        "  unique_words = len(uniques_set)\n",
        "\n",
        "  # Devuelve X, y y unique_words por separado:\n",
        "  return X, y, unique_words"
      ],
      "metadata": {
        "id": "7TkhQGpc7wvd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# [A] 🤓 Clasificiación [0: Humano | 1: Máquina]"
      ],
      "metadata": {
        "id": "8hffjg5vJPVU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")"
      ],
      "metadata": {
        "id": "OzGDbzGwPmMM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import pickle\n",
        "from tqdm import tqdm\n",
        "from gensim.models import Word2Vec"
      ],
      "metadata": {
        "id": "66E1d791SX-k"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# [01] 🏡 KNeighborsClassifier"
      ],
      "metadata": {
        "id": "IzFRab5gpBkx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Variaciones de función de obtención"
      ],
      "metadata": {
        "id": "4dTxsyeOSsrf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define la función para cargar embeddings y obtener datos de entrenamiento:\n",
        "def get_train_data_and_embeddings(dataset_info, size=1):\n",
        "    if dataset_info[\"embeddings\"][\"own\"]:\n",
        "        embeds_model, word2Index, embeds_csv = load_embeddings(\n",
        "            dataset_info[\"embeddings\"][\"model\"],\n",
        "            dataset_info[\"embeddings\"][\"word2Index\"],\n",
        "            dataset_info[\"embeddings\"][\"dataset\"]\n",
        "        )\n",
        "        embeddings, word2Index, embeds_model = load_embeddings(embeds_model, word2Index, embeds_csv)\n",
        "        X, y, unique_words = get_own(dataset_info[\"train_dataset\"], embeds_model)\n",
        "    else:\n",
        "        embeddings = Word2Vec.load(str(dataset_info[\"embeddings\"][\"path\"]))\n",
        "        X, y, unique_words = get_gensim(dataset_info[\"train_dataset\"], embeddings)\n",
        "\n",
        "    return X[:int(len(X)*size)], y[:int(len(y)*size)], unique_words"
      ],
      "metadata": {
        "id": "V_yeYrSeScOS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "72717d42-8968-4fc5-b264-ef00a722f9ca"
      },
      "outputs": [],
      "source": [
        "# Define la función para cargar embeddings y obtener datos de entrenamiento:\n",
        "def get_test_data_and_embeddings(dataset_info, size=1):\n",
        "    if dataset_info[\"embeddings\"][\"own\"]:\n",
        "        embeds_model, word2Index, embeds_csv = load_embeddings(\n",
        "            dataset_info[\"embeddings\"][\"model\"],\n",
        "            dataset_info[\"embeddings\"][\"word2Index\"],\n",
        "            dataset_info[\"embeddings\"][\"dataset\"]\n",
        "        )\n",
        "        embeddings, word2Index, embeds_model = load_embeddings(embeds_model, word2Index, embeds_csv)\n",
        "        X, y, unique_words = get_own(dataset_info[\"test_dataset\"], embeds_model)\n",
        "    else:\n",
        "        embeddings = Word2Vec.load(str(dataset_info[\"embeddings\"][\"path\"]))\n",
        "        X, y, unique_words = get_gensim(dataset_info[\"test_dataset\"], embeddings)\n",
        "\n",
        "    return X[:int(len(X)*size)], y[:int(len(y)*size)], unique_words"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0f81678c-550e-480d-893a-a4879c34703f"
      },
      "source": [
        "## Average pooling"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d2ff86f5-e3bb-4d78-b2ae-ae21bd791bac"
      },
      "outputs": [],
      "source": [
        "from sklearn.neighbors import KNeighborsClassifier"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a20626e7-b065-4073-861c-34f3ecd2adaa"
      },
      "source": [
        "### Baseline - KNN"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "afdda2ea-38bb-4eb7-828a-b03ff7dda0e9"
      },
      "outputs": [],
      "source": [
        "dataset_info = TaskA_datasets[\"A\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "98537cb3-ca5c-4f1d-baf7-6a3c1b32c66b"
      },
      "outputs": [],
      "source": [
        "X, y, a = get_train_data_and_embeddings(dataset_info)\n",
        "avg_pool = [np.mean(sentence, axis=0) for sentence in X]\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(avg_pool, y)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "052bf843-366b-4100-8762-8d3a27f1b4e8"
      },
      "outputs": [],
      "source": [
        "knn = KNeighborsClassifier(metric='cosine')\n",
        "knn.fit(X_train, y_train)\n",
        "y_pred = knn.predict(X_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7c44343b-2419-4098-a271-263303cf427a",
        "outputId": "1693260b-71a4-4f61-bb1b-abc378c02f03"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.77      0.90      0.83     15764\n",
            "           1       0.86      0.70      0.77     14176\n",
            "\n",
            "    accuracy                           0.81     29940\n",
            "   macro avg       0.82      0.80      0.80     29940\n",
            "weighted avg       0.81      0.81      0.80     29940\n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(classification_report(y_test, y_pred))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b04175c0-061b-4d4e-8021-465c212e2092"
      },
      "source": [
        "### Test data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9cc380e9-f7e3-4ad0-b6b5-c5cff0b80263"
      },
      "outputs": [],
      "source": [
        "import seaborn as sns\n",
        "from sklearn.metrics import classification_report\n",
        "from sklearn.metrics import confusion_matrix"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e6509222-90c5-46d3-88bc-c499ebe2fbfb"
      },
      "outputs": [],
      "source": [
        "X_train, y_train = avg_pool, y"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "457e53ec-38cf-46f7-b00a-3e7d333c50c2"
      },
      "outputs": [],
      "source": [
        "X_test, y_test, _ = get_test_data_and_embeddings(dataset_info)\n",
        "X_test = [np.mean(sentence, axis=0) for sentence in X_test]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7b744746-790a-47d2-8bd8-a60dfd0d41b0"
      },
      "outputs": [],
      "source": [
        "knn = KNeighborsClassifier(metric='cosine')\n",
        "knn.fit(X_train, y_train)\n",
        "y_pred = knn.predict(X_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dd05850a-c09b-42fd-836e-60f0fa2c6f65",
        "outputId": "df9e0c64-0de5-448e-93c4-ef297770a638"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.54      0.82      0.65      2500\n",
            "           1       0.63      0.30      0.41      2500\n",
            "\n",
            "    accuracy                           0.56      5000\n",
            "   macro avg       0.58      0.56      0.53      5000\n",
            "weighted avg       0.58      0.56      0.53      5000\n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(classification_report(y_test, y_pred))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ca180c4b-9bfc-409b-ae48-5df39dc1c791"
      },
      "outputs": [],
      "source": [
        "test_path = dir_path + 'TaskA-DevAB.csv'\n",
        "train_path = dir_path + 'TaskA-TrainAB.csv'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a048f576-7b3c-4143-be1f-6c0370b6dc05"
      },
      "outputs": [],
      "source": [
        "test_data = pd.read_csv(test_path)\n",
        "train_data = pd.read_csv(train_path)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9dfff8b7-3150-4350-a8d1-bf4e565d6b76"
      },
      "source": [
        "Results are not stellar, but they DO NOT generalize to the test dataset. The classifier seems to be picking up on patterns that exist in the training data (as a whole) but do not exist in the test data, which gives enough of a reason to think that test data is somehow different from the training data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "13aa6ad9-3ec9-477a-b9c6-e9050d0e51b7"
      },
      "source": [
        "# Experiment"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "416a51d6-5613-4e5b-855d-a5dbb9763d2c"
      },
      "outputs": [],
      "source": [
        "plotdf_train = pd.DataFrame((train_data.text.map(len), y)).T\n",
        "plotdf_train.columns = ['text', 'label']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8f642760-ee4d-43bb-8023-5b383f5bc711"
      },
      "outputs": [],
      "source": [
        "plotdf_test = pd.DataFrame((test_data.text.map(len), y_test)).T\n",
        "plotdf_test.columns = ['text', 'label']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c067afba-5162-4c6e-aeca-62919f355a2c",
        "outputId": "5fa3e7b2-e026-4ade-f6a2-0ac5e9fa9981"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkIAAAHFCAYAAAAe+pb9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXRU9f3/8dcQs7IMhkCQJQRFlBAWCVQDBaJgNFFcqJWiYiJgtYzfliIuiEVAJFUWQ21A0Qp4bCtWgdYWgdQFqICEAEJJRMWEAQmEAXHMwiSZ3N8f/JgakwlJyGQmmefjnDnt/Xw+937e93KUl3c1GYZhCAAAwA+18nYBAAAA3kIQAgAAfosgBAAA/BZBCAAA+C2CEAAA8FsEIQAA4LcIQgAAwG8RhAAAgN8iCAEAAL9FEAL8yL59+/TAAw+oZ8+eCgkJUZs2bTRo0CC98MILOn36tGtcdHS0br31Vi9W2nBLly7VypUrvV2GT1m/fr1mz57t7TIAn0QQAvzEq6++qri4OGVlZemxxx7Thg0btHbtWv385z/Xyy+/rEmTJnm7xEZBEKpu/fr1mjNnjrfLAHzSJd4uAIDnbd++Xb/61a904403at26dQoODnb13XjjjXr00Ue1YcOGJq3J6XSqoqKiSi2+yjAMnT17VqGhod4uBUAj44wQ4Afmz58vk8mk5cuX1xg8goKCdNttt1Vr37BhgwYNGqTQ0FBdffXVev3116v0nzx5UlOmTFFMTIzatGmjTp066YYbbtDWrVurjMvPz5fJZNILL7ygefPmqWfPngoODtZHH32ks2fP6tFHH9XAgQNlNpsVHh6u+Ph4/f3vf69WT2VlpV566SUNHDhQoaGhat++va677jr94x//kHTukt6BAwe0efNmmUwmmUwmRUdHu9a32+2aPn26evbsqaCgIHXt2lVTp05VcXFxlXlMJpMeeeQRvfzyy+rTp4+Cg4O1atWqWo/xX/7yF8XHx6tNmzZq06aNBg4cqD/96U9Vxrz++usaMGCAQkJCFB4erjvvvFO5ublVxiQkJCghIaHa9lNTU6vsy/ljunDhQi1evFg9e/ZUmzZtFB8frx07dlRZLyMjw7Vf53/5+fmSpL/97W+69tprZTabFRYWpssvv1wTJ06sdV+BloQzQkAL53Q69eGHHyouLk7du3ev83qfffaZHn30UT355JOKjIzUa6+9pkmTJqlXr14aMWKEJLnuK3rmmWfUuXNnFRUVae3atUpISNAHH3xQ7S/0P/zhD+rdu7cWLlyodu3a6corr5TD4dDp06c1ffp0de3aVWVlZfr3v/+tsWPHasWKFbr//vtd66empurNN9/UpEmTNHfuXAUFBWn37t2uv9TXrl2ru+66S2azWUuXLpUkV/ArKSnRyJEjdfToUT311FPq37+/Dhw4oFmzZmn//v3697//LZPJ5Jpr3bp12rp1q2bNmqXOnTurU6dObo/VrFmz9Oyzz2rs2LF69NFHZTab9d///leHDx92jUlLS9NTTz2l8ePHKy0tTadOndLs2bMVHx+vrKwsXXnllXX+s/mhjIwMXX311UpPT5ck/e53v1NycrLy8vJkNpv1u9/9TsXFxXrnnXe0fft213qXXXaZtm/frnHjxmncuHGaPXu2QkJCdPjwYX344YcNqgVolgwALdrx48cNScYvfvGLOq/To0cPIyQkxDh8+LCrrbS01AgPDzceeught+tVVFQY5eXlxqhRo4w777zT1Z6Xl2dIMq644gqjrKys1rnPb2PSpEnGNddc42rfsmWLIcmYOXNmrev37dvXGDlyZLX2tLQ0o1WrVkZWVlaV9nfeeceQZKxfv97VJskwm83G6dOna53LMAzj66+/NgICAox7773X7Zhvv/3WCA0NNZKTk6u0W61WIzg42LjnnntcbSNHjqyx/pSUFKNHjx6u5fPHtF+/fkZFRYWrfefOnYYk469//aurzWKxGDX9637hwoWGJOPMmTMX3E+gpeLSGIAaDRw4UFFRUa7lkJAQ9e7du8pZDkl6+eWXNWjQIIWEhOiSSy5RYGCgPvjgg2qXfCTptttuU2BgYLX2v/3tbxo2bJjatGnj2saf/vSnKtt4//33JUkWi6VB+/PPf/5TsbGxGjhwoCoqKly/m266SSaTSR9//HGV8TfccIMuvfTSC243MzNTTqez1rq2b9+u0tJSpaamVmnv3r27brjhBn3wwQcN2SVJ0i233KKAgADXcv/+/SWp2p9TTYYMGSJJuvvuu/X222/rm2++aXAdQHNFEAJauIiICIWFhSkvL69e63Xo0KFaW3BwsEpLS13Lixcv1q9+9Stde+21evfdd7Vjxw5lZWXp5ptvrjLuvMsuu6xa25o1a3T33Xera9euevPNN7V9+3ZlZWVp4sSJOnv2rGvcyZMnFRAQoM6dO9drP847ceKE9u3bp8DAwCq/tm3byjAM2Wy2C9Zak5MnT0qSunXr5nbMqVOn3G6zS5curv6G+PGf0/lLgTUd/x8bMWKE1q1bp4qKCt1///3q1q2bYmNj9de//rXB9QDNDfcIAS1cQECARo0apffff19Hjx6t9S/s+nrzzTeVkJCgZcuWVWn//vvvaxz/w3twfriNnj17avXq1VX6HQ5HlXEdO3aU0+nU8ePH6xxSfigiIkKhoaHVbvj+Yf+Faq1Jx44dJUlHjx51ew/W+bBSUFBQre/YsWNV5g4JCdF3331XbdyPg1pjuf3223X77bfL4XBox44dSktL0z333KPo6GjFx8d7ZE7Al3BGCPADM2bMkGEYevDBB1VWVlatv7y8XO+99169t2symao9hbZv374qN+XWZRtBQUFVgsfx48erPTWWlJQkSdVC14/9+KzVebfeeqsOHTqkDh06aPDgwdV+P3wiqz4SExMVEBBQa13x8fEKDQ3Vm2++WaX96NGj+vDDDzVq1ChXW3R0tL744osqQfDUqVPatm1bg+qT6naWKDg4WCNHjtTzzz8vSdqzZ0+D5wOaE84IAX4gPj5ey5Yt05QpUxQXF6df/epX6tu3r8rLy7Vnzx4tX75csbGxGjNmTL22e+utt+rZZ5/VM888o5EjR+rgwYOaO3euevbsqYqKijpvY82aNZoyZYruuusuHTlyRM8++6wuu+wyffnll65xw4cP14QJEzRv3jydOHFCt956q4KDg7Vnzx6FhYXp//7v/yRJ/fr101tvvaXVq1fr8ssvV0hIiPr166epU6fq3Xff1YgRI/Tb3/5W/fv3V2VlpaxWqzZt2qRHH31U1157bb32XzoXXJ566ik9++yzKi0t1fjx42U2m5WTkyObzaY5c+aoffv2+t3vfqennnpK999/v8aPH69Tp05pzpw5CgkJ0TPPPOPa3oQJE/TKK6/ovvvu04MPPqhTp07phRdeULt27epd23n9+vWTJD3//PNKSkpSQECA+vfvr3nz5uno0aMaNWqUunXrpjNnzmjJkiUKDAzUyJEjGzwf0Kx4+25tAE1n7969RkpKihEVFWUEBQUZrVu3Nq655hpj1qxZRmFhoWtcjx49jFtuuaXa+j9+osnhcBjTp083unbtaoSEhBiDBg0y1q1b5/YJpwULFtRY1+9//3sjOjraCA4ONvr06WO8+uqrxjPPPFPtSSen02m8+OKLRmxsrBEUFGSYzWYjPj7eeO+991xj8vPzjcTERKNt27aGpCp1FBUVGU8//bRx1VVXudbv16+f8dvf/tY4fvy4a5wkw2Kx1PWwGoZhGG+88YYxZMgQIyQkxGjTpo1xzTXXGCtWrKgy5rXXXjP69+/vmvv22283Dhw4UG1bq1atMvr06WOEhIQYMTExxurVq+t1TCUZzzzzjGvZ4XAYkydPNjp27GiYTCZDkpGXl2f885//NJKSkoyuXbsaQUFBRqdOnYzk5GRj69at9dp3oDkzGYZheDGHAQAAeA33CAEAAL9FEAIAAH6LIAQAAPwWQQgAAPgtghAAAPBbBCEAAOC3eKHiBVRWVurYsWNq27ZtnV+5DwAAvMswDH3//ffq0qWLWrVyf96HIHQBx44dc/v9IAAA4NuOHDlS6zcWCUJuZGRkKCMjw/WZgCNHjlzUK+4BAEDTsdvt6t69u9q2bVvrON4sfQF2u11ms1nfffcdQQgAgGairn9/c7M0AADwWwQhAADgtwhCAADAb3GzNAAAXuJ0OlVeXu7tMpqlwMBABQQEXPR2CEIAADQxwzB0/PhxnTlzxtulNGvt27dX586dL+o9fwQhAACa2PkQ1KlTJ4WFhfHC3noyDEMlJSUqLCyUJF122WUN3hZByI3z7xFyOp3eLgUA0II4nU5XCOrQoYO3y2m2QkNDJUmFhYXq1KlTgy+TcbO0GxaLRTk5OcrKyvJ2KQCAFuT8PUFhYWFerqT5O38ML+Y+K4IQAABewOWwi9cYx5AgBAAA/Bb3CAEA4COsVqtsNluTzRcREaGoqKgmm++HoqOjNXXqVE2dOtUr859HEAIAwAdYrVb16XO1SkpKm2zOsLBQ5eZ+XucwlJCQoIEDByo9Pf2i587KylLr1q0vejsXiyAEAIAPsNlsKikp1ZtP3a0+UR09Pl+u9aTum/+2bDZbo50VMgxDTqdTl1xy4XjRsaPn97EuCEIAAPiQPlEdNah3V2+XUU1qaqo2b96szZs3a8mSJZKkFStW6IEHHtCGDRs0c+ZM7du3Txs3blRUVJSmTZumHTt2qLi4WH369FFaWppGjx7t2t6PL42ZTCa9+uqr+te//qWNGzeqa9euWrRokW677TaP7hdByItuSh6jE7VcC46MiNDG9e81YUUAANRsyZIl+uKLLxQbG6u5c+dKkg4cOCBJevzxx7Vw4UJdfvnlat++vY4ePark5GTNmzdPISEhWrVqlcaMGaODBw/WevZpzpw5euGFF7RgwQK99NJLuvfee3X48GGFh4d7bL8IQl50wmZTynOr3PavmpnShNUAAOCe2WxWUFCQwsLC1LlzZ0nS559/LkmaO3eubrzxRtfYDh06aMCAAa7lefPmae3atfrHP/6hRx55xO0cqampGj9+vCRp/vz5eumll7Rz507dfPPNntglSTw+DwAALtLgwYOrLBcXF+vxxx9XTEyM2rdvrzZt2ujzzz+X1WqtdTv9+/d3/f/WrVurbdu2rs9oeApnhNzgExsAANTNj5/+euyxx7Rx40YtXLhQvXr1UmhoqO666y6VlZXVup3AwMAqyyaTSZWVlY1e7w8RhNywWCyyWCyy2+0ym83eLgcAAK8LCgqq0wmCrVu3KjU1VXfeeackqaioSPn5+R6urmG4NAYAAOokOjpan376qfLz82Wz2dyerenVq5fWrFmjvXv36rPPPtM999zj8TM7DcUZIQAAfEiu9aTPzjN9+nSlpKQoJiZGpaWlWrFiRY3jXnzxRU2cOFFDhw5VRESEnnjiCdnt9ost2SMIQgAA+ICIiAiFhYXqvvlvN9mcYWGhioiIqPP43r17a/v27VXaUlNTq42Ljo7Whx9+WKXNYrFUWf7xpTLDMKpt58yZM3WuraEIQgAA+ICoqCjl5n7uN98a8xUEIQAAfERUVJTfB5Omxs3SAADAbxGEAACA3yIIAQAAv0UQAgAAfosgBAAA/BZBCAAA+C2/eXy+pKREffr00c9//nMtXLjQ2+VIkr45nKff3XWd2/7goEDt3r27xj7e/QAAwMXzmyD03HPP6dprr/V2GS5Wq1UV35/U9J8Euh2TvsOhuLi4GvtCw8L0eW4uYQgAWhCr1eo3L1SMjo7W1KlTNXXqVK/Mf55fBKEvv/xSn3/+ucaMGaP//ve/3i5HkmSz2WRUVqr/T29U63aX1jjmzSNbNenRxdXaT1gP6c/PPyabzUYQAoAWwmq16uo+fVRaUtJkc9b3P6oTEhI0cOBApaenX/TcWVlZat269UVv52J5PQht2bJFCxYsUHZ2tgoKCrR27VrdcccdVcYsXbpUCxYsUEFBgfr27av09HQNHz68znNMnz5dCxYs0LZt2xq7/IvWut2latehU419QcEh6nZl3yauCADgDTabTaUlJbr3iQWKjLrC4/N54j+qDcOQ0+nUJZdcOF507NixUea8WF4PQsXFxRowYIAeeOAB/exnP6vWv3r1ak2dOlVLly7VsGHD9MorrygpKUk5OTmuP7i4uDg5HI5q627atElZWVnq3bu3evfu7ZNBCACAH4qMusIn/yM4NTVVmzdv1ubNm7VkyRJJ0ooVK/TAAw9ow4YNmjlzpvbt26eNGzcqKipK06ZN044dO1RcXKw+ffooLS1No0ePdm3vx5fGTCaTXn31Vf3rX//Sxo0b1bVrVy1atEi33XabR/fL60EoKSlJSUlJbvsXL16sSZMmafLkyZKk9PR0bdy4UcuWLVNaWpokKTs72+36O3bs0FtvvaW//e1vKioqUnl5udq1a6dZs2bVON7hcFQJVXa7vSG7BQBAi7JkyRJ98cUXio2N1dy5cyVJBw4ckCQ9/vjjWrhwoS6//HK1b99eR48eVXJysubNm6eQkBCtWrVKY8aM0cGDB2s9+zRnzhy98MILWrBggV566SXde++9Onz4sMLDwz22Xz79+HxZWZmys7OVmJhYpT0xMbHOZ3fS0tJ05MgR5efna+HChXrwwQfdhqDz481ms+vXvXv3i9oHAABaArPZrKCgIIWFhalz587q3LmzAgICJElz587VjTfeqCuuuEIdOnTQgAED9NBDD6lfv3668sorNW/ePF1++eX6xz/+UescqampGj9+vHr16qX58+eruLhYO3fu9Oh++XQQstlscjqdioyMrNIeGRmp48ePe2TOGTNm6LvvvnP9jhw54pF5AABoKQYPHlxlubi4WI8//rhiYmLUvn17tWnTRp9//rmsVmut2+nfv7/r/7du3Vpt27ZVYWGhR2o+z+uXxurCZDJVWTYMo1pbXaSmpl5wTHBwsIKDg+u9bQAA/NWPn/567LHHtHHjRi1cuFC9evVSaGio7rrrLpWVldW6ncDAqq+UMZlMqqysbPR6f8ing1BERIQCAgKqnf0pLCysdpaosWVkZCgjI0NOp9Oj8wAA0FwEBQXV6e/FrVu3KjU1VXfeeackqaioSPn5+R6urmF8+tJYUFCQ4uLilJmZWaU9MzNTQ4cO9ejcFotFOTk5ysrK8ug8AAA0F9HR0fr000+Vn58vm83m9mxNr169tGbNGu3du1efffaZ7rnnHo+f2Wkor58RKioq0ldffeVazsvL0969exUeHu56/G7ChAkaPHiw4uPjtXz5clmtVj388MNerBoAAM84YT3ks/NMnz5dKSkpiomJUWlpqVasWFHjuBdffFETJ07U0KFDFRERoSeeeMJnn8L2ehDatWuXrr/+etfytGnTJEkpKSlauXKlxo0bp1OnTmnu3LkqKChQbGys1q9frx49eni0Li6NAQCaUkREhELDwvTn5x9rsjlDw8IUERFR5/G9e/fW9u3bq7TVdP9tdHS0PvzwwyptFoulyvKPL5UZhlFtO2fOnKlzbQ3l9SCUkJBQ487/0JQpUzRlypQmqugci8Uii8Uiu90us9ncpHMDAPxPVFSUPs/N9ZtvjfkKrwchAABwTlRUlN8Hk6bm0zdLAwAAeBJByI2MjAzFxMRoyJAh3i4FAAB4CEHIDR6fBwCg5SMIAQAAv8XN0l52qrBAJSUlNfaVFhdp7/bN1dc5dtjTZQEA4BcIQm54+j1CDodDkvTFrq1ux3x7wqE3nvmlm16TaxsAAKBhCEJuePo9Quc/7HrDNZerfZuQGse8+3WeMiYkVmvPO/6tfvd6Jh+HBQDgIhGEvOzKrh10WYe2NfZd2rZA944eWK199xff6HevZ9awBgAAqA+CEAAAPuKOWxJ1+uSJJpsvvGOk1v1rU53HJyQkaODAgUpPT2+U+VNTU3XmzBmtW7euUbbXEAQhAAB8xOmTJ7Tl+V802XwjnniryebyVTw+7wYvVAQA4H9SU1O1efNmLVmyRCaTSSaTSfn5+crJyVFycrLatGmjyMhITZgwocr30t555x3169dPoaGh6tChg0aPHq3i4mLNnj1bq1at0t///nfX9j7++OMm3y+CkBu8UBEAgP9ZsmSJ4uPj9eCDD6qgoEAFBQUKDAzUyJEjNXDgQO3atUsbNmzQiRMndPfdd0uSCgoKNH78eE2cOFG5ubn6+OOPNXbsWBmGoenTp+vuu+/WzTff7Nre0KFDm3y/uDQGAAAuyGw2KygoSGFhYercubMkadasWRo0aJDmz5/vGvf666+re/fu+uKLL1RUVKSKigqNHTtWPXr0kCT169fPNTY0NFQOh8O1PW8gCAEAgAbJzs7WRx99pDZt2lTrO3TokBITEzVq1Cj169dPN910kxITE3XXXXfp0ksv9UK1NePSGAAAaJDKykqNGTNGe/furfL78ssvNWLECAUEBCgzM1Pvv/++YmJi9NJLL+mqq65SXl6et0t3IQgBAIA6CQoKqvLFhUGDBunAgQOKjo5Wr169qvxat24tSTKZTBo2bJjmzJmjPXv2KCgoSGvXrq1xe95AEHKDp8YAAKgqOjpan376qfLz82Wz2WSxWHT69GmNHz9eO3fu1Ndff61NmzZp4sSJcjqd+vTTTzV//nzt2rVLVqtVa9as0cmTJ9WnTx/X9vbt26eDBw/KZrOpvLy8yfeJIOQGT40BAFDV9OnTFRAQoJiYGHXs2FFlZWX65JNP5HQ6ddNNNyk2Nla/+c1vZDab1apVK7Vr105btmxRcnKyevfuraefflqLFi1SUlKSJOnBBx/UVVddpcGDB6tjx4765JNPmnyfuFkaAAAfEd4xsklfchjeMbJe43v37q3t27dXa1+zZk2N4/v06aMNGza43V7Hjh21aVPd32ztCQQhAAB8RH0+d4HGwaUxAADgtwhCAADAbxGEAACA3yIIAQAAv0UQcoP3CAEA0PIRhNzgPUIAALR8BCEAAOC3CEIAAMBvEYQAAIDf4s3SAAD4iJuSx+iEzdZk80VGRGjj+vfqPD4hIUEDBw5Uenp6o8yfmpqqM2fOaN26dY2yvYYgCAEA4CNO2GxKeW5Vk823amZKk83lq7g0BgAALig1NVWbN2/WkiVLZDKZZDKZlJ+fr5ycHCUnJ6tNmzaKjIzUhAkTZPvBWa133nlH/fr1U2hoqDp06KDRo0eruLhYs2fP1qpVq/T3v//dtb2PP/64yfeLIAQAAC5oyZIlio+P14MPPqiCggIVFBQoMDBQI0eO1MCBA7Vr1y5t2LBBJ06c0N133y1JKigo0Pjx4zVx4kTl5ubq448/1tixY2UYhqZPn667775bN998s2t7Q4cObfL94tIYAAC4ILPZrKCgIIWFhalz586SpFmzZmnQoEGaP3++a9zrr7+u7t2764svvlBRUZEqKio0duxY9ejRQ5LUr18/19jQ0FA5HA7X9ryBIAQAABokOztbH330kdq0aVOt79ChQ0pMTNSoUaPUr18/3XTTTUpMTNRdd92lSy+91AvV1oxLY27wiQ0AAGpXWVmpMWPGaO/evVV+X375pUaMGKGAgABlZmbq/fffV0xMjF566SVdddVVysvL83bpLgQhN/jEBgAAVQUFBcnpdLqWBw0apAMHDig6Olq9evWq8mvdurUkyWQyadiwYZozZ4727NmjoKAgrV27tsbteQNBCAAA1El0dLQ+/fRT5efny2azyWKx6PTp0xo/frx27typr7/+Wps2bdLEiRPldDr16aefav78+dq1a5esVqvWrFmjkydPqk+fPq7t7du3TwcPHpTNZlN5eXmT7xP3CAEA4CMiIyKa9N0+kRER9Ro/ffp0paSkKCYmRqWlpcrLy9Mnn3yiJ554QjfddJMcDod69Oihm2++Wa1atVK7du20ZcsWpaeny263q0ePHlq0aJGSkpIkSQ8++KA+/vhjDR48WEVFRfroo4+UkJDggT11jyAEAICPqM9bnr2hd+/e2r59e7X2NWvW1Di+T58+2rBhg9vtdezYUZs2bWq0+hqCS2MAAMBvcUbIh/33m2INtPypWnupo1ym4DA98pvfatvWzV6oDACAloEg5MPKjVZK+e1T1doLTtm1/J9ZOnXmgBeqAgCg5eDSGAAA8FsEIQAAvMAwDG+X0Ow1xjEkCAEA0IQCAwMlSSUlJV6upPk7fwzPH9OG4B4hAACaUEBAgNq3b6/CwkJJUlhYmEwmk5eral4Mw1BJSYkKCwvVvn17BQQENHhbBCEAAJrY+a+tnw9DaJj27dtf9Jfr/SIIXXLJJYqNjZUkDR48WK+99pqXKwIA+DOTyaTLLrtMnTp18spnJVqCwMDAizoTdJ5fBKH27dtr79693i4DAIAqAgICGuUvczQcN0sDAAC/5fUgtGXLFo0ZM0ZdunSRyWTSunXrqo1ZunSpevbsqZCQEMXFxWnr1q31msNutysuLk4//elPtXkzb2IGAADneP3SWHFxsQYMGKAHHnhAP/vZz6r1r169WlOnTtXSpUs1bNgwvfLKK0pKSlJOTo6ioqIkSXFxcXI4HNXW3bRpk7p06aL8/Hx16dJF//3vf3XLLbdo//79ateuncf3DQAA+DavB6GkpCQlJSW57V+8eLEmTZqkyZMnS5LS09O1ceNGLVu2TGlpaZKk7OzsWufo0qWLJCk2NlYxMTH64osvNHjw4BrHOhyOKqHKbrfXa38AAEDz4fVLY7UpKytTdna2EhMTq7QnJiZq27ZtddrGt99+6wo2R48eVU5Oji6//HK349PS0mQ2m12/7t27N3wHAACAT/PpIGSz2eR0OhUZGVmlPTIyUsePH6/TNnJzczV48GANGDBAt956q5YsWaLw8HC342fMmKHvvvvO9Tty5MhF7QMAAPBdXr80Vhc/fuOmYRh1fgvn0KFDtX///jrPFRwcrODg4HrVBwAAmiefPiMUERGhgICAamd/CgsLq50lamwZGRmKiYnRkCFDPDoPAADwHp8OQkFBQYqLi1NmZmaV9szMTA0dOtSjc1ssFuXk5CgrK8uj8wAAAO/x+qWxoqIiffXVV67lvLw87d27V+Hh4YqKitK0adM0YcIEDaXqjjoAACAASURBVB48WPHx8Vq+fLmsVqsefvhhL1YNAABaAq8HoV27dun66693LU+bNk2SlJKSopUrV2rcuHE6deqU5s6dq4KCAsXGxmr9+vXq0aOHR+vKyMhQRkaGnE6nR+cBAADe4/UglJCQIMMwah0zZcoUTZkypYkqOsdischischut8tsNjfp3AAAoGn49D1CAAAAnkQQAgAAfosg5AaPzwMA0PJ5/R4hX9Uc7hFynHVo9+7d9V4vIiLC9cFaAAD8GUGoGSoqLZMkWY9YFRcXV+/1Q8PC9HluLmEIAOD3CELN0NmyCklSu/BITXp6Ub3WPWE9pD8//5hsNhtBCADg9whCbjSH9wgFBAaq25V9vV0GAADNFkHIjeZwj9D3p09q0a9/UWNfG/OleujZZU1cEQAAzQtBqJkbNjWjxvZP0i1NXAkAAM0Pj88DAAC/RRACAAB+iyDkBi9UBACg5SMIuWGxWJSTk6OsrCxvlwIAADyEIAQAAPwWQQgAAPgtghAAAPBbBCEAAOC3CEJu8NQYAAAtH0HIDZ4aAwCg5SMIAQAAv0UQAgAAfosgBAAA/BZBCAAA+C2CEAAA8FsEIQAA4LcIQm7wHiEAAFo+gpAbvEcIAICWjyAEAAD81iXeLgCecargiBb9+hfV2svLHDIFh+mR3/xW27Zu9kJlAAD4DoJQC1VpGBo2NaNau/3USW1fv1qnzhzwQlUAAPgWLo0BAAC/RRACAAB+iyAEAAD8FkEIAAD4LYIQAADwWwQhAADgtwhCbvCJDQAAWj6CkBt8YgMAgJaPIAQAAPwWQQgAAPgtghAAAPBbBCEAAOC3CEIAAMBvEYQAAIDfIggBAAC/RRACAAB+iyAEAAD81iXeLgDe4Tjr0O7du+u9XkREhKKiojxQEQAATY8g5GccpcWSJKv1sAYPHV7jGMNZIVWU1dgXGhamz3NzCUMAgBbBL4JQXl6eJk6cqBMnTiggIEA7duxQ69atvV2WV5SXOyRJAYHBGv3UqhrH7P7TDN336Lxq7Sesh/Tn5x+TzWYjCAEAWgS/CEKpqamaN2+ehg8frtOnTys4ONjbJXmdyWRSuw4da+wLDApWtyv7NnFFAAA0vRYfhA4cOKDAwEANH37uMlB4eLiXKwIAAL6iQU+N5eXlNVoBW7Zs0ZgxY9SlSxeZTCatW7eu2pilS5eqZ8+eCgkJUVxcnLZu3Vrn7X/55Zdq06aNbrvtNg0aNEjz589vtNoBAEDz1qAg1KtXL11//fV68803dfbs2YsqoLi4WAMGDNAf//jHGvtXr16tqVOnaubMmdqzZ4+GDx+upKQkWa1W15i4uDjFxsZW+x07dkzl5eXaunWrMjIytH37dmVmZiozM/OiagYAAC1Dgy6NffbZZ3r99df16KOP6pFHHtG4ceM0adIk/eQnP6n3tpKSkpSUlOS2f/HixZo0aZImT54sSUpPT9fGjRu1bNkypaWlSZKys7Pdrt+tWzcNGTJE3bt3lyQlJydr7969uvHGG2sc73A45HA4XMt2u73e+wQAAJqHBp0Rio2N1eLFi/XNN99oxYoVOn78uH7605+qb9++Wrx4sU6ePNkoxZWVlSk7O1uJiYlV2hMTE7Vt27Y6bWPIkCE6ceKEvv32W1VWVmrLli3q06eP2/FpaWkym82u3/kABQAAWp6LerP0JZdcojvvvFNvv/22nn/+eR06dEjTp09Xt27ddP/996ugoOCiirPZbHI6nYqMjKzSHhkZqePHj9e5xvnz52vEiBHq37+/rrzySt16661ux8+YMUPfffed63fkyJGL2gcAAOC7LuqpsV27dun111/XW2+9pdatW2v69OmaNGmSjh07plmzZun222/Xzp07L7pIk8lUZdkwjGpttbnQ5bcfCg4O5vF6AAD8RIOC0OLFi7VixQodPHhQycnJeuONN5ScnKxWrc6dYOrZs6deeeUVXX311RdVXEREhAICAqqd/SksLKx2lqixZWRkKCMjQ06n06Pz+KJTBUe06Ne/qNZeXuaQKThMj/zmt9q2dbMXKgMAoHE1KAgtW7ZMEydO1AMPPKDOnTvXOCYqKkp/+tOfLqq4oKAgxcXFKTMzU3feeaerPTMzU7fffvtFbftCLBaLLBaL7Ha7zGazR+fyNZWGoWFTM6q120+d1Pb1q3XqzAEvVAUAQONrUBDKzMxUVFSU6wzQeYZh6MiRI4qKilJQUJBSUlIuuK2ioiJ99dVXruW8vDzt3btX4eHhioqK0rRp0zRhwgQNHjxY8fHxWr58uaxWqx5++OGGlA4AAODSoCB0xRVXqKCgQJ06darSfvr0afXs2bNel5N27dql66+/3rU8bdo0SVJKSopWrlypcePG6dSpU5o7d64KCgoUGxur9evXq0ePHg0pvc78+dIYAAD+okFByDCMGtuLiooUEhJSr20lJCS43d55U6ZM0ZQpU+q13Yvlz5fGAADwF/UKQufP1phMJs2aNUthYWGuPqfTqU8//VQDBw5s3AoBAAA8pF5BaM+ePZLOnRHav3+/goKCXH1BQUEaMGCApk+f3rgVAgAAeEi9gtBHH30kSXrggQe0ZMkStWvXziNF+QLuEQIAoOVr0JulV6xY0aJDkHTuHqGcnBxlZWV5uxQAAOAhdT4jNHbsWK1cuVLt2rXT2LFjax27Zs2aiy4MAADA0+ochMxms+uzFjxFBQAAWoI6B6EVK1bU+P9bKu4RAgCg5WvQPUKlpaUqKSlxLR8+fFjp6enatGlToxXmbdwjBABAy9egFyrefvvtGjt2rB5++GGdOXNGP/nJTxQUFCSbzabFixfrV7/6VWPXiWbkpuQxOmGz1dgXGRGhjevfa+KKAACoWYOC0O7du/Xiiy9Kkt555x117txZe/bs0bvvvqtZs2YRhPzcCZtNKc+tqrFv1cwLf38OAICm0qBLYyUlJWrbtq0kadOmTRo7dqxatWql6667TocPH27UAgEAADylQUGoV69eWrdunY4cOaKNGzcqMTFRklRYWNhi3i+UkZGhmJgYDRkyxNulAAAAD2lQEJo1a5amT5+u6OhoXXvttYqPj5d07uzQNddc06gFegs3SwMA0PI16B6hu+66Sz/96U9VUFCgAQMGuNpHjRqlO++8s9GKAwAA8KQGBSFJ6ty5szp37lyl7Sc/+clFFwQAANBUGhSEiouL9fvf/14ffPCBCgsLVVlZWaX/66+/bpTi4JscZx3avXu32/7S0lIVFBRUaw8LC/NkWQAA1FuDgtDkyZO1efNmTZgwQZdddpnr0xto2RylxZIk6xGr4uLi3I4zBYdp+fLl1doDAwPVqazcY/UBAFBfDQpC77//vv71r39p2LBhjV2Pz+ATG9WVlzskSe3CIzXp6UVux7256GkNSh5Xpa3Iflr7/5OpCmeFR2sEAKA+GhSELr30UoWHhzd2LT7FYrHIYrHIbrfzkdkfCQgMVLcr+7rtDwwKVrsOHZuwIgAAGqZBj88/++yzmjVrVpXvjQEAADQ3DTojtGjRIh06dEiRkZGKjo5WYGBglf7abqQFAADwFQ0KQnfccUdj1wEAANDkGhSEnnnmmcauAwAAoMk16B4hSTpz5oxee+01zZgxQ6dPn5Z07pLYN99802jFAQAAeFKDzgjt27dPo0ePltlsVn5+vh588EGFh4dr7dq1Onz4sN54443GrhMAAKDRNSgITZs2TampqXrhhRfUtm1bV3tSUpLuueeeRivOm5rDe4QMw9DxI4dr7qusrLHvu1M2T5cFAECz0aAglJWVpVdeeaVae9euXXX8+PGLLsoX+PJ7hJz//5MmTmeFPvv4vRrHVFSUu+0DAADnNCgIhYSEyG63V2s/ePCgOnbkRXqeFtDq3K1dgQGtNHZ4zS82/Mtn/6qxz1p4RrsOch8XAABSA4PQ7bffrrlz5+rtt9+WJJlMJlmtVj355JP62c9+1qgFwj2TyaR+l3eusS+gVSu3fQQhAADOadBTYwsXLtTJkyfVqVMnlZaWauTIkerVq5fatm2r5557rrFrBAAA8IgGnRFq166d/vOf/+ijjz5Sdna2KisrNWjQII0ePbqx6wMAAPCYegehyspKrVy5UmvWrFF+fr5MJpN69uypzp07yzAMmUwmT9QJAADQ6Op1acwwDN12222aPHmyvvnmG/Xr1099+/bV4cOHlZqaqjvvvNNTdQIAADS6ep0RWrlypbZs2aIPPvhA119/fZW+Dz/8UHfccYfeeOMN3X///Y1aJAAAgCfUKwj99a9/1VNPPVUtBEnSDTfcoCeffFJ//vOfCUJwy3HWod27d9d7vYiICEVFRXmgIgCAP6tXENq3b59eeOEFt/1JSUn6wx/+cNFFoeVxlBZLkqxHrIqLi6v3+qFhYfo8N5cwBABoVPUKQqdPn1ZkZKTb/sjISH377bcXXZQvaA6f2GhOyssdkqR24ZGa9PSieq17wnpIf37+MdlstlqD0E3JY3TC5v4TIpEREdq4nrdtAwD+p15ByOl06pJL3K8SEBCgioqKiy7KF/jyJzaas4DAQHW7sua3YV+sEzabUp5b5bZ/1cwUj8wLAGi+6hWEDMNQamqqgoODa+x3OByNUhQAAEBTqFcQSkm58H9Rc6M0AABoLuoVhFasWOGpOgAAAJpcg741BgAA0BIQhAAAgN8iCAEAAL/VoK/Pw799f/qkFv36F277T534xmNz1/auoLy8fI/NCwBomQhCaJBhUzPc9q17fIzH5q3tXUFP3/1Tj80LAGiZuDQGAAD8FmeE0Gzk5uaqtLRUBQUFNfZXVla67QsLC/NkaQCAZoogBJ9nP31SknTffffJFBym5cuX1zjOUeZw2xcYGKhOZeUeqxEA0Dy1+CB08OBBjRs3rsryX//6V91xxx1erAr1UVpklyTd8tBMZWWu1aDkcTWO+2DfBsXX0FdkP639/8lUhbNlfAcPANB4WnwQuuqqq7R3715JUlFRkaKjo3XjjTd6uSo0RIcuPRQYFKx2HTrW2G8ymdz2AQBQE7+6Wfof//iHRo0apdatW3u7FAAA4AO8HoS2bNmiMWPGqEuXLjKZTFq3bl21MUuXLlXPnj0VEhKiuLg4bd26tUFzvf3221UukwEAAP/m9SBUXFysAQMG6I9//GON/atXr9bUqVM1c+ZM7dmzR8OHD1dSUpKsVqtrTFxcnGJjY6v9jh075hpjt9v1ySefKDk52eP7BAAAmgev3yOUlJSkpKQkt/2LFy/WpEmTNHnyZElSenq6Nm7cqGXLliktLU2SlJ2dfcF5/v73v+umm25SSEhIreMcDoccDodr2W6312U3AABAM+T1IFSbsrIyZWdn68knn6zSnpiYqG3bttVrW2+//bZ++ctfXnBcWlqa5syZU69te0pFmUOzpk+r1u6srFR5hVNGpVFjvySVV1R6urwGudDnOdqYL9VDzy5rworOqe3THZIUGRGhjevfa8KK6qa51g0AvsKng5DNZpPT6VRkZGSV9sjISB0/frzO2/nuu++0c+dOvfvuuxccO2PGDE2b9r9wYbfb1b1797oX3YjCg5365sWh1dr3fX1ca7bm6OXdFTqenlDjum2n1i8oNqXaPs/xSbqlCSv5n9o+3SFJq2amNGE1dddc6wYAX+HTQeg8k8lUZdkwjGpttTGbzTpx4kSdxgYHBys4OLhe9QEAgObJ6zdL1yYiIkIBAQHVzv4UFhZWO0vU2DIyMhQTE6MhQ4Z4dB4AAOA9Ph2EgoKCFBcXp8zMzCrtmZmZGjq0+iWjxmSxWJSTk6OsrCyPzgMAALzH65fGioqK9NVXX7mW8/LytHfvXoWHhysqKkrTpk3ThAkTNHjwYMXHx2v58uWyWq16+OGHvVg1AABoCbwehHbt2qXrr7/etXz+RuWUlBStXLlS48aN06lTpzR37lwVFBQoNjZW69evV48ePbxVMgAAaCG8HoQSEhJkGEatY6ZMmaIpU6Y0UUXnZGRkKCMjQ06ns0nnhec4zjq0e/fuGvtKS0tVUFBQY19YWJgnywIAeJHXg5CvslgsslgsstvtMpvN3i4HF8FRWixJsh6xKi4ursYxpuAwLV++vMa+wMBAdSor91h9AADvIQihxSsvP/em8HbhkZr09KIax7y56GkNSq7+Hboi+2nt/0+mKpwVHq0RAOAdBCE3uDTW8gQEBqrblX1r7AsMCla7Dh2buCIAgLf59OPz3sTj8wAAtHwEIQAA4LcIQgAAwG8RhNzgExsAALR8BCE3uEcIAICWjyAEAAD8FkEIAAD4LYIQAADwW7xQ0Q1eqOgdpwqOaNGvf1GlrbTILlNwmP75p4Uq+f6MlypDS3BT8hidsNnc9kdGRGjj+veasCIA3kYQcoNvjXlHpWFo2NSMKm3H8g5q/38yFZswRjteesRLlaElOGGzKeW5VW77V81MacJqAPgCLo0BAAC/RRACAAB+i0tjQB04zjq0e/fueq8XERGhqKgoD1QEAGgMBCGgFo7SYkmS9YhVcXFx9V4/NCxMn+fmEoYAwEcRhIBalJc7JEntwiM16elF9Vr3hPWQ/vz8Y7LZbAQhAPBRBCE3eHwePxQQGKhuV/b1dhkAgEbGzdJu8K0xAABaPoIQAADwWwQhAADgtwhCAADAbxGEAACA3yIIAQAAv0UQAgAAfosg5EZGRoZiYmI0ZMgQb5cCAAA8hCDkBu8RAgCg5ePN0oCPslqtstlstY4pLS1VQUFBtfawsDCZzWZPlQYALQZBCPBBVqtVV/fpo9KSklrHmYLDtHz58mrtgYGBslgsnioPAFoMghDgg2w2m0pLSnTvEwsUGXWF23FvLnpag5LHVWkrsp/W/v9kquQCIQoAQBACfFpk1BW1fuw1MChY7Tp0bMKKAKBl4WZpAADgtwhCAADAb3FpDPCw3NzcJlkHAFB/BCHAQ+ynT0qS7rvvvgZvo6ioqLHKAQDUgCAEeEhpkV2SdMtDM3VV/7h6rZu7c7PeX7VEZ8+e9URpAID/jyDkRkZGhjIyMuR0Or1dCpq5Dl161PrkV01OWA95qBoAwA9xs7QbfGIDAICWjyAEAAD8FpfGgBbKZrOptLRUu3fvrtd6ERERioqK8lBVAOBbCEJAC+MoLZYkrVmzRvYvv1RcXP1u1A4NC9PnubmEIQB+gSAEtDDl5Q5J0pWDh8tasFf3PTqvzuuesB7Sn59/TDabjSAEwC8QhIAWKqxNewUGBdf7iTUA8CfcLA0AAPwWQQgAAPgtghAAAPBbBCEAAOC3uFkaQDW5ubkNWo93EAFobghCAFzsp09Kku67774Grc87iAA0N34RhF588UW99tprMgxDo0eP1pIlS2QymbxdFuBzSovskqRbHpqpq/rX70WMvIMIQHPU4oPQyZMn9cc//lEHDhxQYGCgRowYoR07dig+Pt7bpQE+q0OXHrx/CIBfaPFBSJIqKip09uxZSVJ5ebk6derk5YoAAIAv8PpTY1u2bNGYMWPUpUsXmUwmrVu3rtqYpUuXqmfPngoJCVFcXJy2bt1a5+137NhR06dPV1RUlLp06aLRo0friiuuaMxdAAAAzZTXg1BxcbEGDBigP/7xjzX2r169WlOnTtXMmTO1Z88eDR8+XElJSbJara4xcXFxio2NrfY7duyYvv32W/3zn/9Ufn6+vvnmG23btk1btmxpqt0DAAA+zOuXxpKSkpSUlOS2f/HixZo0aZImT54sSUpPT9fGjRu1bNkypaWlSZKys7Pdrv+3v/1NvXr1Unh4uCTplltu0Y4dOzRixIgaxzscDjkcDtey3W6v9z4BAIDmwetBqDZlZWXKzs7Wk08+WaU9MTFR27Ztq9M2unfvrm3btuns2bMKDAzUxx9/rF/+8pdux6elpWnOnDkXVTfgz3gHEYDmxKeDkM1mk9PpVGRkZJX2yMhIHT9+vE7buO6665ScnKxrrrlGrVq10qhRo3Tbbbe5HT9jxgxNmzbNtWy329W9e/eG7QDgR3gHEYDmyKeD0Hk/fuePYRj1eg/Qc889p+eee65OY4ODgxUcHFyv+gDwDiIAzZNPB6GIiAgFBARUO/tTWFhY7SxRY8vIyFBGRoacTqdH5wFaGt5BBKA58fpTY7UJCgpSXFycMjMzq7RnZmZq6NChHp3bYrEoJydHWVlZHp0HAAB4j9fPCBUVFemrr75yLefl5Wnv3r0KDw9XVFSUpk2bpgkTJmjw4MGKj4/X8uXLZbVa9fDDD3uxagAA0BJ4PQjt2rVL119/vWv5/I3KKSkpWrlypcaNG6dTp05p7ty5KigoUGxsrNavX68ePXp4tC4ujQEA0PJ5PQglJCTIMIxax0yZMkVTpkxpoorOsVgsslgsstvtMpvNTTo3AABoGj59jxAAAIAnEYTcyMjIUExMjIYMGeLtUgAAgIcQhNzgqTEAAFo+r98jBADn8XkOAE2NIATA6/g8BwBvIQgB8Do+zwHAWwhCbvAeIaDp8XkOAE2Nm6Xd4GZpAABaPoIQAADwWwQhAADgtwhCAADAbxGE3ODN0gAAtHwEITe4WRoAgJaPIAQAAPwW7xEC0CLU5fMcpaWlKigoqNIWFhYms9nsqbIA+DiCEIBmrT6f5zAFh2n58uVV2gIDA2WxWAhDgJ8iCAFo1urzeY43Fz2tQcnjXMtF9tPa/59MlZSUEIQAP0UQcoNPbADNS10+zxEYFKx2HTo2UUUAmgNulnaDp8YAAGj5CEIAAMBvEYQAAIDfIggBAAC/xc3SANBAVqtVNputQetGREQoKiqqkSsCUF8EIQBoAKvVqqv79FFpSUmD1g8NC9PnubmEIcDLCEIA0AA2m02lJSW694kFioy6ol7rnrAe0p+ff0w2m40gBHgZQcgN3iMEoC4io6644PuLAPgubpZ2g/cIAQDQ8hGEAACA3yIIAQAAv0UQAgAAfosgBAAA/BZBCAAA+C2CEAAA8Fu8R8hPGYah40cO19xXWVlj33enGvYpAcDXnf9MRmlpqXbv3l2ndXJzcz1ZksfwWRCgKoKQn3FWVp77X2eFPvv4vRrHVFSUu+0DWhJHabEkac2aNZIk+5dfKi4url7bKCoqavS6PIXPggDVEYT8TECrc1dDAwNaaezwmt+G+5fP/lVjn7XwjHYd/Maj9QFNqbzcIUm6cvBwRXTqot0Fe3Xfo/PqtG7uzs16f9USnT171pMlNio+CwJURxByo6V/YsNkMqnf5Z1r7Ato1cptH0EILVFYm/Zq16GjAoOC6/y5jBPWQx6uynP4LAjwP9ws7Qaf2AAAoOUjCAEAAL9FEAIAAH6LIAQAAPwWQQgAAPgtghAAAPBbBCEAAOC3CEIAAMBvEYQAAIDfIggBAAC/RRACAAB+iyAEAAD8ll8EoYULF6pv376KjY3Vm2++6e1yAACAj2jxX5/fv3+//vKXvyg7O1uSNGrUKN16661q3769lysDAADe1uLPCOXm5mro0KEKCQlRSEiIBg4cqA0bNni7LAAA4AO8fkZoy5YtWrBggbKzs1VQUKC1a9fqjjvuqDJm6dKlWrBggQoKCtS3b1+lp6dr+PDhddp+bGys5syZozNnzkiSPvzwQ11++eWNvh8AAPesVqtsNluD1o2IiFBUVFQjV+S7muOxao41n+f1IFRcXKwBAwbogQce0M9+9rNq/atXr9bUqVO1dOlSDRs2TK+88oqSkpKUk5PjOnBxcXFyOBzV1t20aZNiYmL061//WjfccIPMZrOGDBmiSy7x+m4DgN+wWq26uk8flZaUNGj90LAwfZ6b6xdhqDkeq+ZY8w95PREkJSUpKSnJbf/ixYs1adIkTZ48WZKUnp6ujRs3atmyZUpLS5Mk1/0/7jz00EN66KGHJEmTJ09Wr1693I51OBxVQpXdbq/zvgAAqrPZbCotKdG9TyxQZNQV9Vr3hPWQ/vz8Y7LZbH4RhJrjsWqONf+Q14NQbcrKypSdna0nn3yySntiYqK2bdtW5+0UFhaqU6dOOnjwoHbu3KmXX37Z7di0tDTNmTOnwTUDAGoWGXWFul3Z19tlNAvN8Vg1x5olHw9CNptNTqdTkZGRVdojIyN1/PjxOm/njjvu0JkzZ9S6dWutWLGi1ktjM2bM0LRp01zLdrtd3bt3r3/xAADA5/l0EDrPZDJVWTYMo1pbbepz9ig4OFjBwcF1Hg8AAJovn358PiIiQgEBAdXO/hQWFlY7S9TYMjIyFBMToyFDhnh0HgAA4D0+HYSCgoIUFxenzMzMKu2ZmZkaOnSoR+e2WCzKyclRVlaWR+cBAADe4/VLY0VFRfrqq69cy3l5edq7d6/Cw8MVFRWladOmacKECRo8eLDi4+O1fPlyWa1WPfzww16sGgAAtAReD0K7du3S9ddf71o+f6NySkqKVq5cqXHjxunUqVOaO3euCgoKFBsbq/Xr16tHjx4erSsjI0MZGRlyOp0enQcAAHiP14NQQkKCDMOodcyUKVM0ZcqUJqroHIvFIovFIrvdLrPZ3KRzAwCApuHT9wgBAAB4EkEIAAD4LYKQGzw+DwBAy0cQcoPH5wEAaPkIQgAAwG8RhAAAgN/y+uPzvur8e4QqKioknfv4amMqKiqSYUjfn61Q69LyGsdUGobsNfQVOZxyVBhu+6Vz32M7W1parb3McVZGRbnb/rquW15a7HbfDKOyWn+Fo/Si161wlNTYX9u6VeaudOpscVGN61Y6nQ1e151yx1lJUtnZUo+tW1PdPzxelc761d0UNXtz3R8frx8eq/LS4nodr4up2VFaIuncvwca+98ttSkqKnLN35Q1e2ve5qg5Hitfrfn89i70ih6TcaERfu7o0aN8fR4AgGbqyJEj6tatm9t+gtAFVFZW6tixY2rbtm29vnh/IXa7Xd27d9eRI0fUrl27Rtsu6o4/A+/i+HsXx9+7OP6eZxiGvv/+e3Xp0kWtWrm/E4hLYxfQqlWrWpPkxWrXrh3/RF6O/wAACuRJREFUEHgZfwbexfH3Lo6/d3H8PasuX4bgZmkAAOC3CEIAAMBvBcyePXu2t4vwVwEBAUpISNAll3CF0lv4M/Aujr93cfy9i+PvG7hZGgAA+C0ujQEAAL9FEAIAAH6LIAQAAPwWQQgAAPgtgpCXLF26VD179lRISIji4uK0detWb5fk02bPni2TyVTl17lzZ1e/YRiaPXu2unTpotDQUCUkJOjAgQNVtuFwOPR///d/ioiIUOvWrXXbbbfp6NGjVcZ8++23mjBhgsxms8xmsyZMmKAzZ85UGWO1WjVmzBi1bt1aERER+vWvf62ysjLP7byXbNmyRWPGjFGXLl1kMpm0bt26Kv2+dsz379+vkSNHKjQ0VF27dtXcuXMv+I0hX3ah45+amlrtn4nrrruuyhiOf8OlpaVpyJAhatu2rTp16qQ77rhDBw8erDKGfwZaCANN7q233jICAwONV1991cjJyTF+85vfGK1btzYOHz7s7dJ81jPPPGP07dvXKCgocP0KCwtd/b///e+Ntm3bGu+++66xf/9+Y9y4ccZll11m2O1215iHH37Y6Nr1/7V37yFN/X0cwN9qbpTZSNLNZakEEnGs0MiM7oFmGVH/lIRZQVG06EoFRUV0hYz6o4huUhRFoEFQlJqXLlsX1EqzSLJVlGZeGkWluX1+fzx0fixnD0/Pcmvn/QJhO+ezs+95j6/7MPc9DpSioiKprKyUyZMny4gRI6Szs1OtmTZtmiiKIlarVaxWqyiKIpmZmer+zs5OURRFJk+eLJWVlVJUVCRms1ksFkvPBNGDrl69Kps3b5b8/HwBIJcuXXLb70+ZOxwOMRqNMm/ePKmurpb8/HwJDw+X/fv3/8GE/qz/ln9OTo5MmzbNbU60tLS41TD/35eeni55eXlSU1MjDx8+lBkzZsjgwYPl8+fPag3nQGBgI+QDo0ePlmXLlrltGzp0qGzatMlHI/J/27ZtkxEjRnjc53K5xGQyyd69e9Vt3759E4PBIEePHhURkY8fP0poaKhcuHBBrXn79q0EBwfLtWvXRESktrZWAMjdu3fVGpvNJgDk2bNnIvKfN6fg4GB5+/atWnP+/HnR6/XicDi8d8J+5uc3Yn/L/MiRI2IwGOTbt29qzZ49e8RsNovL5fJmFD7RXSM0a9asbh/D/L2rqalJAEh5ebmIcA4EEv5prId1dHSgoqICaWlpbtvT0tJgtVp9NKq/Q11dHcxmM+Lj4zFv3jzU19cDAF6+fInGxka3TPV6PSZOnKhmWlFRge/fv7vVmM1mKIqi1thsNhgMBqSkpKg1Y8aMgcFgcKtRFAVms1mtSU9PR3t7OyoqKv7cyfsZf8vcZrNh4sSJ0Ov1bjXv3r2D3W73fgB+oqysDFFRUUhISMCSJUvQ1NSk7mP+3uVwOAAAERERADgHAgkboR7W3NwMp9MJo9Hott1oNKKxsdFHo/J/KSkpOHPmDK5fv47jx4+jsbERY8eORUtLi5rbrzJtbGyETqdD//79f1kTFRXV5bmjoqLcan5+nv79+0On02nq9fO3zD3V/LgfqK9LRkYGzp07h5KSEuTm5uLBgweYMmUK2tvbATB/bxIRrF27FuPGjYOiKAA4BwIJr+vtI0FBQW73RaTLNvpXRkaGejsxMRGpqakYMmQITp8+rX5B9Hcy/bnGU/3v1GiFP2XuaSzdPTYQzJ07V72tKApGjRqF2NhYXLlyBXPmzOn2ccz/f2exWPD48WPcvn27yz7Ogb8fPxHqYQMGDEBISEiXDr2pqalLN0/dCwsLQ2JiIurq6tTVY7/K1GQyoaOjA21tbb+sef/+fZfn+vDhg1vNz8/T1taG79+/a+r187fMPdX8+DORVl6X6OhoxMbGoq6uDgDz95aVK1fi8uXLKC0tRUxMjLqdcyBwsBHqYTqdDsnJySgqKnLbXlRUhLFjx/poVH+f9vZ2PH36FNHR0YiPj4fJZHLLtKOjA+Xl5WqmycnJCA0NdatpaGhATU2NWpOamgqHw4H79++rNffu3YPD4XCrqampQUNDg1pTWFgIvV6P5OTkP3rO/sTfMk9NTcXNmzfdlhMXFhbCbDYjLi7O+wH4oZaWFrx58wbR0dEAmP//S0RgsVhQUFCAkpISxMfHu+3nHAggPfrVbBKRf5fPnzx5Umpra2X16tUSFhYmdrvd10PzW+vWrZOysjKpr6+Xu3fvSmZmpoSHh6uZ7d27VwwGgxQUFEh1dbVkZWV5XMYaExMjxcXFUllZKVOmTPG4jHX48OFis9nEZrNJYmKix2WsU6dOlcrKSikuLpaYmJiAXD7/6dMnqaqqkqqqKgEgBw4ckKqqKvUyD/6U+cePH8VoNEpWVpZUV1dLQUGB9OvX769eOvyr/D99+iTr1q0Tq9UqL1++lNLSUklNTZWBAwcyfy9Zvny5GAwGKSsrc7tEwZcvX9QazoHAwEbIRw4fPiyxsbGi0+kkKSlJXZJJnv24PkdoaKiYzWaZM2eOPHnyRN3vcrlk27ZtYjKZRK/Xy4QJE6S6utrtGF+/fhWLxSIRERHSu3dvyczMlNevX7vVtLS0yPz58yU8PFzCw8Nl/vz50tbW5lbz6tUrmTFjhvTu3VsiIiLEYrG4LVkNFKWlpQKgy09OTo6I+F/mjx8/lvHjx4terxeTySTbt2//q5cN/yr/L1++SFpamkRGRkpoaKgMHjxYcnJyumTL/H+fp+wBSF5enlrDORAYgkR42UkiIiLSJn5HiIiIiDSLjRARERFpFhshIiIi0iw2QkRERKRZbISIiIhIs9gIERERkWaxESIiIiLNYiNEREREmsVGiIgCxqRJk7B69Wq/PyYR+Q82QkRERKRZbISIKCAsXLgQ5eXlOHToEIKCghAUFAS73Y7a2lpMnz4dffv2hdFoRHZ2NpqbmwEAZWVl0Ol0uHXrlnqc3NxcDBgwAA0NDd0ek4gCB//XGBEFBIfDgYyMDCiKgh07dgAAnE4nRo4ciSVLlmDBggX4+vUrNm7ciM7OTpSUlAAANmzYgIsXL+LRo0ew2+1ISUnB+fPnMXv2bI/HjIyMREhIiM/Ok4i8q5evB0BE5A0GgwE6nQ59+vSByWQCAGzduhVJSUnYvXu3Wnfq1CkMGjQIz58/R0JCAnbu3Ini4mIsXboUT548QXZ2NmbPnt3tMYkosLARIqKAVVFRgdLSUvTt27fLvhcvXiAhIQE6nQ5nz57F8OHDERsbi4MHD/pgpETkK2yEiChguVwuzJw5E/v27euyLzo6Wr1ttVoBAK2trWhtbUVYWFiPjZGIfIuNEBEFDJ1OB6fTqd5PSkpCfn4+4uLi0KuX5193L168wJo1a3D8+HFcvHgRCxYswI0bNxAcHOzxmEQUWLhqjIgCRlxcHO7duwe73Y7m5masWLECra2tyMrKwv3791FfX4/CwkIsXrwYTqcTTqcT2dnZSEtLw6JFi5CXl4eamhrk5uZ2e0yXy+XDMyQib2MjREQBY/369QgJCcGwYcMQGRmJjo4O3LlzB06nE+np6VAUBatWrYLBYEBwcDB27doFu92OY8eOAQBMJhNOnDiBLVu24OHDhx6P+fr1a1+eIhF5GZfPExERkWbxEyEiIiLSLDZCREREpFlshIiIiEiz2AgRERGRZrERIiIiIs1iI0RERESaxUaIiIiINIuNEBEREWkWGyEiIiLSLDZCREREpFlshIiIiEiz2AgRERGRZv0DQRH0pH/q2PUAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sns.histplot(data=plotdf_train, x='text', stat='density', bins=30, label='train',\n",
        "             multiple='stack', hue='label', alpha=0.5)\n",
        "sns.histplot(data=plotdf_test, x='text', stat='density', bins=30, label='test',\n",
        "             multiple='stack', hue='label', alpha=0.5)\n",
        "plt.title('Character counts')\n",
        "plt.legend()\n",
        "plt.yscale('log')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ffedf0b-e303-4871-8daf-11f8e540d362"
      },
      "source": [
        "this is kind of cheating, but this should help for these specific datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "adaeb0eb-2092-4a14-be47-8d54bbba3afb"
      },
      "source": [
        "## Second attempt - add 1-grams"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "034cceed-3fcc-4d39-b99d-1fce07bbf607"
      },
      "outputs": [],
      "source": [
        "X, y, _ = get_train_data_and_embeddings(dataset_info)\n",
        "counts = np.array([len(sentence) for sentence in X])\n",
        "avg_pool = np.array([np.mean(sentence, axis=0) for sentence in X])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "407011fe-ac23-4ffe-9716-5a8798cd7056"
      },
      "outputs": [],
      "source": [
        "counts = counts.reshape(-1, 1)\n",
        "counts = StandardScaler().fit_transform(counts)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7104774a-b22d-487c-b9f7-36a8a65a722f"
      },
      "outputs": [],
      "source": [
        "X = np.hstack([counts, avg_pool])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a4d15efe-b5c5-4819-97c9-e17c2370918c",
        "outputId": "403324db-470f-4069-872c-e5882b432a90",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 177
        }
      },
      "outputs": [
        {
          "output_type": "error",
          "ename": "NameError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-1-462c3120ceb9>\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_test\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_test_split\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m: name 'train_test_split' is not defined"
          ]
        }
      ],
      "source": [
        "X_train, X_test, y_train, y_test = train_test_split(X, y)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d9ca1816-bded-4bb2-8e1f-41952ca7a561"
      },
      "outputs": [],
      "source": [
        "knn = KNeighborsClassifier(metric='cosine')\n",
        "knn.fit(X_train, y_train)\n",
        "y_pred = knn.predict(X_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "55b4911f-80e6-424d-895a-eb3cd56b966a",
        "outputId": "f736b36d-dbf1-4180-a436-7e2550777e70"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.81      0.88      0.84     15739\n",
            "           1       0.85      0.77      0.81     14201\n",
            "\n",
            "    accuracy                           0.83     29940\n",
            "   macro avg       0.83      0.82      0.82     29940\n",
            "weighted avg       0.83      0.83      0.83     29940\n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(classification_report(y_test, y_pred))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2bab0d7c-b662-449f-ae56-b61d1bd1d77f"
      },
      "source": [
        "### Test data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8a9cea47-f0d4-4b9c-bd96-a88eefe302e7"
      },
      "outputs": [],
      "source": [
        "X_train, y_train = X, y"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a5a8d76d-7edc-4725-a153-c2aba57f8e5f"
      },
      "outputs": [],
      "source": [
        "X_test, y_test, _ = get_test_data_and_embeddings(dataset_info)\n",
        "counts = np.array([len(sentence) for sentence in X_test]).reshape(-1, 1)\n",
        "\n",
        "counts = StandardScaler().fit_transform(counts)\n",
        "\n",
        "avg_pool = np.array([np.mean(sentence, axis=0) for sentence in X_test])\n",
        "X_test = np.hstack([counts, avg_pool])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0c528495-fb77-474c-b367-0bb05f472c2b"
      },
      "outputs": [],
      "source": [
        "knn = KNeighborsClassifier(metric='cosine')\n",
        "knn.fit(X_train, y_train)\n",
        "y_pred = knn.predict(X_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d9b53d84-b457-4b88-94b3-464ad7858ac8",
        "outputId": "4146ed71-0692-47aa-ad2e-fa208d6f87c3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.55      0.83      0.66      2500\n",
            "           1       0.65      0.32      0.43      2500\n",
            "\n",
            "    accuracy                           0.58      5000\n",
            "   macro avg       0.60      0.58      0.55      5000\n",
            "weighted avg       0.60      0.58      0.55      5000\n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(classification_report(y_test, y_pred))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "be0cc169-d1cb-4a08-bdfe-4d0933090dc5"
      },
      "source": [
        "Better performance overall, but results are not getting closer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "19531eb5-1f42-4621-9198-ed802893e768"
      },
      "source": [
        "Ideas: Try different preprocessing approach, try adding Max Pooling, add features. There's not much point trying to optimize a classifier for the data as it is."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "dataset_info = TaskA_datasets[\"B\"]\n",
        "\n",
        "X, y, _ = get_train_data_and_embeddings(dataset_info)\n",
        "counts = np.array([len(sentence) for sentence in X])\n",
        "avg_pool = np.array([np.mean(sentence, axis=0) for sentence in X])\n",
        "\n",
        "X = np.hstack([counts, avg_pool])\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y)\n",
        "\n",
        "knn = KNeighborsClassifier(metric='cosine')\n",
        "knn.fit(X_train, y_train)\n",
        "y_pred = knn.predict(X_test)\n",
        "\n",
        "print(classification_report(y_test, y_pred))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W5653SOjNYwO",
        "outputId": "5f3046bd-1b2e-49ba-86b4-790a7f329e6a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.79      0.76      0.78     15739\n",
            "           1       0.75      0.77      0.76     14201\n",
            "\n",
            "    accuracy                           0.77     29940\n",
            "   macro avg       0.77      0.77      0.77     29940\n",
            "weighted avg       0.77      0.77      0.77     29940\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "dataset_info = TaskA_datasets[\"C\"]\n",
        "\n",
        "X, y, _ = get_train_data_and_embeddings(dataset_info)\n",
        "counts = np.array([len(sentence) for sentence in X])\n",
        "avg_pool = np.array([np.mean(sentence, axis=0) for sentence in X])\n",
        "\n",
        "X = np.hstack([counts, avg_pool])\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y)\n",
        "\n",
        "knn = KNeighborsClassifier(metric='cosine')\n",
        "knn.fit(X_train, y_train)\n",
        "y_pred = knn.predict(X_test)\n",
        "\n",
        "print(classification_report(y_test, y_pred))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0C7hX8CWGCHq",
        "outputId": "193b2d5d-7478-496b-aca6-a99043d1905f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.88      0.70      0.78     15739\n",
            "           1       0.73      0.90      0.81     14201\n",
            "\n",
            "    accuracy                           0.80     29940\n",
            "   macro avg       0.81      0.80      0.79     29940\n",
            "weighted avg       0.81      0.80      0.79     29940\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "dataset_info = TaskA_datasets[\"D\"]\n",
        "\n",
        "X, y, _ = get_train_data_and_embeddings(dataset_info)\n",
        "counts = np.array([len(sentence) for sentence in X])\n",
        "avg_pool = np.array([np.mean(sentence, axis=0) for sentence in X])\n",
        "\n",
        "X = np.hstack([counts, avg_pool])\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y)\n",
        "\n",
        "knn = KNeighborsClassifier(metric='cosine')\n",
        "knn.fit(X_train, y_train)\n",
        "y_pred = knn.predict(X_test)\n",
        "\n",
        "print(classification_report(y_test, y_pred))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lKlRdF7oRezJ",
        "outputId": "c9738acb-5f03-4358-ec6f-87b91c49ef33"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.79      0.67      0.73     15739\n",
            "           1       0.69      0.80      0.74     14201\n",
            "\n",
            "    accuracy                           0.73     29940\n",
            "   macro avg       0.74      0.74      0.73     29940\n",
            "weighted avg       0.74      0.73      0.73     29940\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# [02] ⏩ RED NEURONAL RECURRENTE"
      ],
      "metadata": {
        "id": "w627pBNygWmw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Modelo de la RED"
      ],
      "metadata": {
        "id": "ezeyPEhMtDMf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "from sklearn.model_selection import StratifiedKFold\n",
        "from sklearn.metrics import accuracy_score, f1_score\n",
        "from sklearn.model_selection import StratifiedKFold, train_test_split\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences"
      ],
      "metadata": {
        "id": "sBMksbSCsxcc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def train_rnn(X_train, y_train, X_test, y_test, epochs, batch_size):\n",
        "\n",
        "  # Pad sequences to a fixed length (adjust maxlen as needed):\n",
        "  max_sequence_length_train = max(len(seq) for seq in X_train)\n",
        "  max_sequence_length_test = max(len(seq) for seq in X_test)\n",
        "\n",
        "  # Use the maximum of the two for padding:\n",
        "  maxlen = max(max_sequence_length_train, max_sequence_length_test)\n",
        "\n",
        "  X_train_padded = pad_sequences(X_train, dtype=\"float32\", padding=\"post\", truncating=\"post\", maxlen=maxlen)\n",
        "  X_test_padded = pad_sequences(X_test, dtype=\"float32\", padding=\"post\", truncating=\"post\", maxlen=maxlen)\n",
        "\n",
        "  y_train = np.expand_dims(y_train, axis=-1)\n",
        "\n",
        "  # Convert NumPy arrays to TensorFlow tensors:\n",
        "  X_train_tensor = tf.convert_to_tensor(X_train_padded, dtype=tf.float32)\n",
        "  y_train_tensor = tf.convert_to_tensor(y_train, dtype=tf.float32)\n",
        "\n",
        "  train_dataset = tf.data.Dataset.from_tensor_slices((X_train_tensor, y_train_tensor)).batch(batch_size)\n",
        "\n",
        "  model = tf.keras.Sequential()\n",
        "  model.add(tf.keras.layers.LSTM(100, input_shape=(maxlen, X_train_padded.shape[2])))\n",
        "  model.add(tf.keras.layers.Dense(1, activation=\"sigmoid\"))\n",
        "\n",
        "  model.compile(loss=\"binary_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])\n",
        "\n",
        "  # Split the training data into training and validation sets:\n",
        "  X_train_split, X_test_split, y_train_split, y_test_split = train_test_split(\n",
        "      X_train_padded, y_train, test_size=0.3, random_state=11\n",
        "  )\n",
        "\n",
        "  history = model.fit(train_dataset, epochs=epochs, validation_data=(X_test_split, y_test_split), verbose=1)\n",
        "\n",
        "  return model, history, maxlen\n",
        "\n",
        "\n",
        "def evaluate_rnn(model, X_test, y_test, maxlen):\n",
        "  # Convert X_test to a NumPy array if it's a list\n",
        "  if isinstance(X_test, list):\n",
        "    X_test = np.array([np.array(x) for x in X_test])\n",
        "\n",
        "  # Pad sequences to a fixed length used during training (adjust maxlen as needed):\n",
        "  X_test_padded = pad_sequences(X_test, dtype=\"float32\", padding=\"post\", truncating=\"post\", maxlen=maxlen)\n",
        "\n",
        "  # Convert X_test to a TensorFlow tensor:\n",
        "  X_test_tensor = tf.convert_to_tensor(X_test_padded, dtype=tf.float32)\n",
        "\n",
        "  # Predict on the test data:\n",
        "  y_pred = model.predict(X_test_tensor)\n",
        "\n",
        "  # Convert predicted probabilities to binary values:\n",
        "  y_pred_binary = (y_pred > 0.5).astype(int)\n",
        "\n",
        "  # Evaluate performance metrics:\n",
        "  accuracy = accuracy_score(y_test, y_pred_binary)\n",
        "  micro_f1 = f1_score(y_test, y_pred_binary, average=\"micro\")\n",
        "  macro_f1 = f1_score(y_test, y_pred_binary, average=\"macro\")\n",
        "\n",
        "  return accuracy, micro_f1, macro_f1"
      ],
      "metadata": {
        "id": "Qs7K-CUhs3XN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# PARA TODOS LOS DATASETS:\n",
        "xepochs = 10\n",
        "xbatch = 64\n",
        "xsize = 0.01\n",
        "print(\"\\n══════════════════════════════════════════\")\n",
        "print(\"\\nCLASIFICADOR DE RED NEURONAL RECURRENTE\")\n",
        "print(\"\\n══════════════════════════════════════════\")\n",
        "for dataset_name, dataset_info in TaskA_datasets.items():\n",
        "  print(f\"\\nDATASET {dataset_name}:\")\n",
        "\n",
        "  # Obtiene datos y embeddings:\n",
        "  X, y, unique_words = get_data_and_embeddings(dataset_info, size=xsize)\n",
        "\n",
        "  # X = np.array(X)\n",
        "  # y = np.array(y)\n",
        "\n",
        "  # Inicializa k-fold cross-validation:\n",
        "  k_folds = 5\n",
        "  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=11)\n",
        "\n",
        "  avg_accuracy = 0\n",
        "  avg_macro_f1 = 0\n",
        "  avg_micro_f1 = 0\n",
        "\n",
        "  skf = StratifiedKFold(n_splits=k_folds, shuffle=True, random_state=11)\n",
        "  for train_index, test_index in skf.split(X, y):\n",
        "\n",
        "    # train_index = np.array(train_index, dtype=int)\n",
        "    # test_index = np.array(test_index, dtype=int)\n",
        "\n",
        "    X_train, X_test = np.array(X)[train_index], np.array(X)[test_index]\n",
        "    y_train, y_test = np.array(y)[train_index], np.array(y)[test_index]\n",
        "\n",
        "    # Entrena el modelo en los datos de entrenamiento:\n",
        "    model, history, maxlen = train_rnn(X_train, y_train, X_test, y_test, epochs=xepochs, batch_size=xbatch)\n",
        "\n",
        "    # Evalua el modelo en los datos de prueba:\n",
        "    accuracy, micro_f1, macro_f1 = evaluate_rnn(model, X_test, y_test, maxlen)\n",
        "    print(\"\\nPara el FOLD:\")\n",
        "    print(f\" * Accuracy: {accuracy*100:.2f}%\\n * F1-Score Micro: {micro_f1*100:.2f}%\\n * F1-Score Macro: {macro_f1*100:.2f}%\\n\")\n",
        "\n",
        "    avg_accuracy += accuracy\n",
        "    avg_macro_f1 += macro_f1\n",
        "    avg_micro_f1 += micro_f1\n",
        "\n",
        "  avg_accuracy /= k_folds\n",
        "  avg_macro_f1 /= k_folds\n",
        "  avg_micro_f1 /= k_folds\n",
        "\n",
        "\n",
        "  # Imprime resultados:\n",
        "  print(f\" * Promedio Accuracy: {avg_accuracy*100:.2f}%\")\n",
        "  print(f\" * Promedio Macro F1: {avg_macro_f1*100:.2f}%\")\n",
        "  print(f\" * Promedio Micro F1: {avg_micro_f1*100:.2f}%\")\n",
        "\n",
        "print(\"\\n══════════════════════════════════════════\")"
      ],
      "metadata": {
        "id": "fgDtigqCmCJM",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aa582fcc-19da-4248-de55-e8d7cf01cb7b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "══════════════════════════════════════════\n",
            "\n",
            "CLASIFICADOR DE RED NEURONAL RECURRENTE\n",
            "\n",
            "══════════════════════════════════════════\n",
            "\n",
            "DATASET A:\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-3-f0c5682550fd>:17: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n",
            "  X = np.array([[word_embeddings[word] for word in sentence.split() if word in word_embeddings]\n",
            "<ipython-input-12-47eeda10dc50>:31: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n",
            "  X_train, X_test = np.array(X)[train_index], np.array(X)[test_index]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# [04] 👬 Autores\n",
        "## Realizado por:\n",
        "* León Rosas Manuel Alejandro.\n",
        "* Ramos Herrera Iván Alejandro."
      ],
      "metadata": {
        "id": "4DinJz64wzft"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## [0X] 🔎 FAILURE SELF-ATTENTION"
      ],
      "metadata": {
        "id": "OUbZYGRMZP0i"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Código del Modelo"
      ],
      "metadata": {
        "id": "Vog7KK0_hbPF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Previo"
      ],
      "metadata": {
        "id": "NyGZjrod_7ro"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import copy\n",
        "import math"
      ],
      "metadata": {
        "id": "csunkQ4YaleX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Positional Encoder:\n",
        "class PositionalEncoding(nn.Module):\n",
        "  def __init__(self, dimension, max_len=1000):\n",
        "    super(PositionalEncoding, self).__init__()\n",
        "\n",
        "    # Matriz de dimensión de embeddings por número máximo de elementos:\n",
        "    pe = torch.zeros(max_len, dimension)\n",
        "    # Posición de 0 al máximo número de elementos (enteros):\n",
        "    position = torch.arange(0, max_len).unsqueeze(1)\n",
        "    # Factores de log y exp:\n",
        "    div_term = torch.exp(torch.arange(0, dimension, 2) * -(math.log(10000.0) / dimension))\n",
        "\n",
        "    # Entradas Pares:\n",
        "    pe[:, 0::2] = torch.sin(position * div_term)\n",
        "    # Entradas Impares:\n",
        "    pe[:, 1::2] = torch.cos(position * div_term)\n",
        "    self.register_buffer(\"pe\", pe)\n",
        "\n",
        "\n",
        "  def forward(self, x):\n",
        "    # Suma del x más su codificación posicional:\n",
        "    # x = x + torch.autograd.Variable(self.pe[:x.size(0), :x.size(1)], requires_grad=False).reshape(x.shape)\n",
        "    x = x + self.pe[:x.size(1), :].unsqueeze(0)\n",
        "\n",
        "    return x"
      ],
      "metadata": {
        "id": "Mehq1AtMlotd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Capa de Normalización:\n",
        "class NormalizationLayer(nn.Module):\n",
        "  def __init__(self, features, eps=1e-6):\n",
        "    super(NormalizationLayer, self).__init__()\n",
        "    self.a_2 = nn.Parameter(torch.ones(features))\n",
        "    self.b_2 = nn.Parameter(torch.zeros(features))\n",
        "    self.eps = eps\n",
        "\n",
        "  def forward(self, x):\n",
        "    mean = x.mean(-1, keepdim=True)\n",
        "    std = x.std(-1, keepdim=True)\n",
        "\n",
        "    # Distribución N(0,1):\n",
        "    return self.a_2 * (x - mean) / (std + self.eps) + self.b_2"
      ],
      "metadata": {
        "id": "1ubhoVhQk2ZL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Capa Auto-Atencional:\n",
        "class AttentionLayer(nn.Module):\n",
        "  def __init__(self, dimension):\n",
        "    super(AttentionLayer, self).__init__()\n",
        "\n",
        "    # Pesos Query:\n",
        "    self.Q = nn.Linear(dimension, dimension, bias=False)\n",
        "    # Pesos Key:\n",
        "    self.K  = nn.Linear(dimension, dimension, bias=False)\n",
        "    # Pesos Value:\n",
        "    self.V  = nn.Linear(dimension, dimension, bias=False)\n",
        "    self.dimension = dimension\n",
        "\n",
        "\n",
        "  def forward(self, x):\n",
        "    # Capas atencionales:\n",
        "    query, key, value = self.Q(x), self.K(x), self.V(x)\n",
        "    print(f\"Query shape: {query.shape}\")\n",
        "    print(f\"Key shape: {key.shape}\")\n",
        "    print(f\"Value shape: {value.shape}\")\n",
        "\n",
        "    # Computa el método de self-attention para las x's totalmente conectadas:\n",
        "    # scores = torch.matmul(query, key.T)/math.sqrt(self.dimension)\n",
        "    scores = torch.matmul(query, key.permute(0, 2, 1)) / math.sqrt(self.dimension)\n",
        "    print(f\"Scores shape: {scores.shape}\")\n",
        "\n",
        "    # Salida con Softmax:\n",
        "    p_attn = torch.nn.functional.softmax(scores, dim = -1)\n",
        "    Vs = torch.matmul(p_attn, value).reshape(x.shape)\n",
        "    print(f\"Output shape after attention: {Vs.shape}\")\n",
        "\n",
        "    return Vs, p_attn"
      ],
      "metadata": {
        "id": "Yh2EJI5MnUWa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# MODELO ATENCIONAL:\n",
        "class AttentionalClassificator(nn.Module):\n",
        "  def __init__(self, vocabulary_size, embed_dimension=50, classes=2):\n",
        "    super(AttentionalClassificator, self).__init__()\n",
        "\n",
        "    # Capa de embeddings:\n",
        "    self.embeddings = nn.Embedding(vocabulary_size, embed_dimension)\n",
        "    # Capa de Positional Encoding:\n",
        "    self.positional = PositionalEncoding(embed_dimension)\n",
        "    # Capa Atencional:\n",
        "    self.attention = AttentionLayer(embed_dimension)\n",
        "    # Capa de Normalización:\n",
        "    self.normalization = NormalizationLayer(embed_dimension)\n",
        "    # Capa de Clasificación FeedForward:\n",
        "    if classes == 2:\n",
        "      self.feedforward = nn.Sequential(nn.Linear(embed_dimension, embed_dimension), nn.Tanh(), nn.Linear(embed_dimension, classes), nn.Sigmoid())\n",
        "    else:\n",
        "      self.feedforward = nn.Sequential(nn.Linear(embed_dimension, embed_dimension), nn.Tanh(), nn.Linear(embed_dimension, classes), nn.Softmax(classes))\n",
        "    # Matriz de Pesos Atencionales:\n",
        "    self.attention_matrix = None\n",
        "    self.dropout = nn.Dropout(0.5)\n",
        "\n",
        "\n",
        "  def forward(self, x):\n",
        "    print(f\"\\nInput shape (before embeddings): {x.shape}\")\n",
        "    # x = x.squeeze(0)\n",
        "    x = x.long()\n",
        "    x_emb = self.positional(self.embeddings(x.squeeze(0)))\n",
        "    print(f\"Embeddings shape: {x_emb.shape}\")\n",
        "    x_emb = self.normalization(x_emb)\n",
        "    h, self.attention_matrix = self.attention(x_emb)\n",
        "    print(f\"Output shape after attention: {h.shape}\")\n",
        "    h = self.normalization(x_emb + h)\n",
        "    h = self.dropout(h)\n",
        "    output = self.feedforward(h)\n",
        "    print(f\"Final output shape: {output.shape}\")\n",
        "    return output\n",
        "\n",
        "    return output"
      ],
      "metadata": {
        "id": "40_Jq9MooCzH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### NUEVA IMPLEMENTACIÓN"
      ],
      "metadata": {
        "id": "p4_3A1YDyjf9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "```python\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "\n",
        "class SelfAttentionClassifier(nn.Module):\n",
        "  def __init__(self, embedding_dim, hidden_dim, num_heads, output_dim, dropout=0.1):\n",
        "    super(SelfAttentionClassifier, self).__init__()\n",
        "\n",
        "    # Input Embedding:\n",
        "    self.embedding = nn.Embedding(unique_words, embedding_dim)\n",
        "\n",
        "    # Positional Embedding:\n",
        "    self.positional_encoding = PositionalEncoding(embedding_dim)\n",
        "\n",
        "    # Multi-Head Attention:\n",
        "    self.multihead_attention = nn.MultiheadAttention(embed_dim=embedding_dim, num_heads=num_heads, dropout=dropout)\n",
        "\n",
        "    # FeedForward:\n",
        "    self.feedforward = nn.Sequential(\n",
        "      nn.Linear(embedding_dim, hidden_dim),\n",
        "      nn.ReLU(),\n",
        "      nn.Dropout(dropout),\n",
        "      nn.Linear(hidden_dim, output_dim)\n",
        "    )\n",
        "\n",
        "  def forward(self, x):\n",
        "    x = x.transpose(0, 1)\n",
        "    x = self.embedding(x)\n",
        "    x = self.positional_encoding(x)\n",
        "    x, _ = self.multihead_attention(x, x, x)\n",
        "    x = x + x\n",
        "    x = nn.LayerNorm(x.size()[1:]).to(\"cpu\")(x)\n",
        "    x = self.feedforward(x)\n",
        "    return x\n",
        "\n",
        "# Clase para la codificación posicional:\n",
        "class PositionalEncoding(nn.Module):\n",
        "  def __init__(self, d_model, max_len=512):\n",
        "    super(PositionalEncoding, self).__init__()\n",
        "    self.encoding = torch.zeros(max_len, d_model)\n",
        "    position = torch.arange(0, max_len).unsqueeze(1).float()\n",
        "    div_term = torch.exp(torch.arange(0, d_model, 2).float() * -(math.log(10000.0) / d_model))\n",
        "    self.encoding[:, 0::2] = torch.sin(position * div_term)\n",
        "    self.encoding[:, 1::2] = torch.cos(position * div_term)\n",
        "    self.encoding = self.encoding.unsqueeze(0)\n",
        "\n",
        "  def forward(self, x):\n",
        "    print(f\"Positional Encoding Input Shape: {x.shape}\")\n",
        "    return x + self.encoding[:, :x.size(1)].detach().to(\"cpu\")\n",
        "```"
      ],
      "metadata": {
        "id": "JW1U2ZB8fkyy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Prueba rápida\n",
        "unique_words = 10000\n",
        "\n",
        "modelx = SelfAttentionClassifier(256, 128, 8, 1)\n",
        "dummy_input = torch.randint(0, unique_words, (32, 64))\n",
        "output = modelx(dummy_input)\n",
        "print(\"dummy_input.shape:\", dummy_input.shape)"
      ],
      "metadata": {
        "id": "snSwahOSgK65"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "torch.randint(0, unique_words, (32, 64)).shape"
      ],
      "metadata": {
        "id": "_xHq161MtZP2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Implementación & Evaluación"
      ],
      "metadata": {
        "id": "-zkIZLo-hg3a"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.utils.data import DataLoader, TensorDataset\n",
        "from sklearn.metrics import accuracy_score, f1_score\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "from sklearn.utils import shuffle\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import math"
      ],
      "metadata": {
        "id": "ISVrhmud4Fgl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Obtiene el dataset de palabras:\n",
        "X, y, unique_words = get_data_and_words(\"/content/drive/MyDrive/Datasets/AITextification/TaskA-TrainAB.csv\", size=0.0001)\n",
        "# Cargar word2Index desde archivo pickle:\n",
        "import pickle\n",
        "with open(\"/content/drive/MyDrive/Datasets/AITextification/B_word2Index.pkl\", \"rb\") as word2Index_file:\n",
        "  word2Index = pickle.load(word2Index_file)\n",
        "# Obtiene índices de palabras a partir de word2Index:\n",
        "X = [[word2Index[word] for word in texto[0].split()] for texto in X]"
      ],
      "metadata": {
        "id": "kRl1hmC20fdD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from keras.preprocessing.sequence import pad_sequences\n",
        "\n",
        "# Encuentra la longitud de la cadena más larga\n",
        "max_length = max(len(seq) for seq in X)\n",
        "print(\"Secuencia más grande:\", max_length)\n",
        "\n",
        "# Rellena las secuencias para que todas tengan la misma longitud\n",
        "X = pad_sequences(X, maxlen=max_length, padding=\"post\", truncating=\"post\", value=-1)\n",
        "\n",
        "# Imprime la nueva forma de X\n",
        "print(\"Nueva forma de X:\", X.shape)\n",
        "print(\"Acá ya todas las secuencias tienen la misma longitud.\")\n",
        "print(\"Tamaño de de y:\", len(y))\n",
        "print(\"Tamaño del vocabulario:\", unique_words)"
      ],
      "metadata": {
        "id": "y82y3mzqT589"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "\n",
        "class MultiHeadAttention(nn.Module):\n",
        "    def __init__(self, d_model, n_heads):\n",
        "        super(MultiHeadAttention, self).__init__()\n",
        "        self.n_heads = n_heads\n",
        "        self.d_model = d_model\n",
        "\n",
        "        self.query_linear = nn.Linear(d_model, d_model)\n",
        "        self.key_linear = nn.Linear(d_model, d_model)\n",
        "        self.value_linear = nn.Linear(d_model, d_model)\n",
        "        self.out_linear = nn.Linear(d_model, d_model)\n",
        "\n",
        "    def forward(self, query, key, value, mask=None):\n",
        "        batch_size, seq_len, d_model = query.size()\n",
        "\n",
        "        # Linear transformations\n",
        "        query = self.query_linear(query).view(batch_size, seq_len, self.n_heads, -1)\n",
        "        key = self.key_linear(key).view(batch_size, seq_len, self.n_heads, -1)\n",
        "        value = self.value_linear(value).view(batch_size, seq_len, self.n_heads, -1)\n",
        "\n",
        "        # Transpose for multi-head attention\n",
        "        query = query.transpose(1, 2)\n",
        "        key = key.transpose(1, 2)\n",
        "        value = value.transpose(1, 2)\n",
        "\n",
        "        # Scaled Dot-Product Attention\n",
        "        scores = torch.matmul(query, key.transpose(-2, -1)) / torch.sqrt(d_model)\n",
        "        if mask is not None:\n",
        "            scores = scores.masked_fill(mask == 0, -1e9)\n",
        "\n",
        "        attention_weights = F.softmax(scores, dim=-1)\n",
        "        attended_values = torch.matmul(attention_weights, value)\n",
        "\n",
        "        # Transpose and concatenate to get the original shape\n",
        "        attended_values = attended_values.transpose(1, 2).contiguous().view(batch_size, seq_len, -1)\n",
        "\n",
        "        # Linear transformation for the output\n",
        "        output = self.out_linear(attended_values)\n",
        "\n",
        "        return output\n",
        "\n",
        "class PositionalEncoding(nn.Module):\n",
        "    def __init__(self, d_model, max_len=512):\n",
        "        super(PositionalEncoding, self).__init__()\n",
        "        self.encoding = torch.zeros(max_len, d_model)\n",
        "        position = torch.arange(0, max_len).unsqueeze(1).float()\n",
        "        div_term = torch.exp(torch.arange(0, d_model, 2).float() * -(math.log(10000.0) / d_model))\n",
        "        self.encoding[:, 0::2] = torch.sin(position * div_term)\n",
        "        self.encoding[:, 1::2] = torch.cos(position * div_term)\n",
        "        self.encoding = self.encoding.unsqueeze(0)\n",
        "\n",
        "    def forward(self, x):\n",
        "        return x + self.encoding[:, :x.size(1)].detach()\n",
        "\n",
        "class TransformerClassifier(nn.Module):\n",
        "    def __init__(self, vocab_size, d_model, n_heads, hidden_dim, num_classes, max_len=100):\n",
        "        super(TransformerClassifier, self).__init__()\n",
        "        self.embedding = nn.Embedding(vocab_size, d_model)\n",
        "        self.positional_encoding = PositionalEncoding(d_model, max_len)\n",
        "        self.multihead_attention = MultiHeadAttention(d_model, n_heads)\n",
        "        self.feedforward = nn.Sequential(\n",
        "            nn.Linear(d_model, hidden_dim),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(hidden_dim, num_classes)\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.embedding(x)\n",
        "        x = self.positional_encoding(x)\n",
        "        attention_output = self.multihead_attention(x, x, x)\n",
        "        x = x + attention_output  # Add & Norm\n",
        "        x = F.layer_norm(x, x.size()[1:])  # Normalization\n",
        "        x = self.feedforward(x)\n",
        "        return F.log_softmax(x, dim=-1)"
      ],
      "metadata": {
        "id": "6wfb1Vn1YVtn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.utils import shuffle\n",
        "\n",
        "# Define la función para cargar embeddings y obtener datos de entrenamiento:\n",
        "def get_data_and_embeddings(dataset_info, size=1):\n",
        "  if dataset_info[\"embeddings\"][\"own\"]:\n",
        "      embeds_model, word2Index, embeds_csv = load_embeddings(\n",
        "          dataset_info[\"embeddings\"][\"model\"],\n",
        "          dataset_info[\"embeddings\"][\"word2Index\"],\n",
        "          dataset_info[\"embeddings\"][\"dataset\"]\n",
        "      )\n",
        "      embeddings, word2Index, embeds_model = load_embeddings(embeds_model, word2Index, embeds_csv)\n",
        "      X, y, unique_words = get_own(dataset_info[\"train_dataset\"], embeds_model)\n",
        "  else:\n",
        "    embeddings = Word2Vec.load(str(dataset_info[\"embeddings\"][\"path\"]))\n",
        "    X, y, unique_words = get_gensim(dataset_info[\"train_dataset\"], embeddings)\n",
        "\n",
        "\n",
        "  # Combina X e y antes de la mezcla aleatoria:\n",
        "  combined = list(zip(X, y))\n",
        "\n",
        "  # Mezcla aleatoria:\n",
        "  combined = shuffle(combined, random_state=11)\n",
        "\n",
        "  # Desempaqueta X e y después de la mezcla aleatoria:\n",
        "  X, y = zip(*combined)\n",
        "\n",
        "  # Devuelve X, y y unique_words por separado:\n",
        "  return X[:int(len(X)*size)], y[:int(len(y)*size)], unique_words"
      ],
      "metadata": {
        "id": "fa-qlhHV8odC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## EJEMPLOS FALLIDOS"
      ],
      "metadata": {
        "id": "LYf9P4w76y1S"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from tqdm import tqdm\n",
        "\n",
        "# Carga los datos y embeddings:\n",
        "dataset_info = TaskA_datasets[\"A\"]\n",
        "X, y, _ = get_data_and_embeddings(dataset_info, size=0.0001)\n",
        "\n",
        "# Divide los datos en conjuntos de entrenamiento y prueba:\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.7, random_state=11)\n",
        "\n",
        "# Rellena las secuencias para que tengan la misma longitud:\n",
        "X_train_padded, X_train_lengths = pad_sequences(X_train)\n",
        "X_test_padded, X_test_lengths = pad_sequences(X_test)\n",
        "\n",
        "# Convierte los datos a tensores de PyTorch:\n",
        "X_train_tensor = torch.tensor(X_train_padded).float()\n",
        "y_train_tensor = torch.tensor(y_train).long()\n",
        "\n",
        "X_test_tensor = torch.tensor(X_test_padded).float()\n",
        "y_test_tensor = torch.tensor(y_test).long()\n",
        "\n",
        "# Crea conjuntos de datos y dataloaders:\n",
        "train_dataset = TensorDataset(X_train_tensor, y_train_tensor)\n",
        "test_dataset = TensorDataset(X_test_tensor, y_test_tensor)\n",
        "\n",
        "train_dataloader = DataLoader(train_dataset, batch_size=32, shuffle=True)\n",
        "test_dataloader = DataLoader(test_dataset, batch_size=32, shuffle=False)\n",
        "\n",
        "# Configura el modelo y el optimizador:\n",
        "model = AttentionalClassificator(vocabulary_size=TaskA_datasets, embed_dimension=50, classes=2)\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
        "\n",
        "# Entrenamiento del modelo:\n",
        "num_epochs = 5\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "model.to(device)\n",
        "\n",
        "for epoch in range(num_epochs):\n",
        "  model.train()\n",
        "  total_loss = 0\n",
        "\n",
        "  for batch_x, batch_y in train_dataloader:\n",
        "    batch_x, batch_y = batch_x.to(device), batch_y.to(device)\n",
        "\n",
        "    optimizer.zero_grad()\n",
        "    outputs = model(batch_x)\n",
        "    loss = criterion(outputs, batch_y)\n",
        "    loss.backward()\n",
        "    optimizer.step()\n",
        "\n",
        "    total_loss += loss.item()\n",
        "\n",
        "# Evaluación del modelo:\n",
        "model.eval()\n",
        "all_preds = []\n",
        "all_labels = []\n",
        "\n",
        "with torch.no_grad():\n",
        "    for batch_x, batch_y in test_dataloader:\n",
        "        batch_x, batch_y = batch_x.to(device), batch_y.to(device)\n",
        "\n",
        "        preds = model(batch_x)\n",
        "        _, pred_labels = torch.max(preds, 1)\n",
        "\n",
        "        all_preds.extend(pred_labels.cpu().numpy())\n",
        "        all_labels.extend(batch_y.cpu().numpy())\n",
        "\n",
        "# Calcula y muestra la accuracy\n",
        "accuracy = accuracy_score(all_labels, all_preds)\n",
        "print(f\"Accuracy on test set: {accuracy}\")"
      ],
      "metadata": {
        "id": "mpDXacLSdW9g"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Función para obtener datos y embeddings:\n",
        "from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence, pad_sequence\n",
        "\n",
        "def get_data_and_attentional_embeddings(dataset_info, max_sequence_length=None):\n",
        "  if dataset_info[\"embeddings\"][\"own\"]:\n",
        "    embeds_model, word2Index, embeds_csv = load_embeddings(\n",
        "        dataset_info[\"embeddings\"][\"model\"],\n",
        "        dataset_info[\"embeddings\"][\"word2Index\"],\n",
        "        dataset_info[\"embeddings\"][\"dataset\"]\n",
        "    )\n",
        "    embeddings, word2Index, embeds_model = load_embeddings(embeds_model, word2Index, embeds_csv)\n",
        "    X, y, unique_words = get_own(dataset_info[\"train_dataset\"], embeds_model, max_sequence_length)\n",
        "  else:\n",
        "    embeddings = Word2Vec.load(str(dataset_info[\"embeddings\"][\"path\"]))\n",
        "    X, y, unique_words = get_gensim(dataset_info[\"train_dataset\"], embeddings, max_sequence_length)\n",
        "\n",
        "  # Aplica Padding a los datos para que tengan la misma longitud:\n",
        "  X_padded = pad_sequences(X, max_sequence_length)\n",
        "\n",
        "  # Convierte a tensores de PyTorch:\n",
        "  X_tensor = torch.tensor(X_padded, dtype=torch.long)\n",
        "  y_tensor = torch.tensor(y, dtype=torch.long)\n",
        "\n",
        "  return X_tensor, y_tensor, unique_words"
      ],
      "metadata": {
        "id": "kQZw_6-SztgP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def pad_sequences(sequences, padding_value=0):\n",
        "  lengths = [len(seq) for seq in sequences]\n",
        "  padded_sequences = pad_sequence([torch.tensor(seq) for seq in sequences], batch_first=True, padding_value=padding_value)\n",
        "  return padded_sequences, lengths"
      ],
      "metadata": {
        "id": "aLGUb_Wm9N8Z"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Carga los datos y embeddings\n",
        "dataset_info = TaskA_datasets[\"A\"]\n",
        "X, y, _ = get_data_and_embeddings(dataset_info)\n",
        "\n",
        "# Divide los datos en conjuntos de entrenamiento y prueba\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=11)\n",
        "\n",
        "# Rellena las secuencias para que tengan la misma longitud\n",
        "X_train_padded, X_train_lengths = pad_sequences(X_train)\n",
        "X_test_padded, X_test_lengths = pad_sequences(X_test)\n",
        "\n",
        "# Convierte los datos a tensores de PyTorch\n",
        "X_train_tensor = torch.tensor(X_train).float()\n",
        "y_train_tensor = torch.tensor(y_train).long()\n",
        "\n",
        "X_test_tensor = torch.tensor(X_test).float()\n",
        "y_test_tensor = torch.tensor(y_test).long()\n",
        "\n",
        "# Crea conjuntos de datos y dataloaders\n",
        "train_dataset = TensorDataset(X_train_tensor, y_train_tensor)\n",
        "test_dataset = TensorDataset(X_test_tensor, y_test_tensor)\n",
        "\n",
        "train_dataloader = DataLoader(train_dataset, batch_size=32, shuffle=True)\n",
        "test_dataloader = DataLoader(test_dataset, batch_size=32, shuffle=False)\n",
        "\n",
        "# Configura el modelo y el optimizador\n",
        "model = AttentionalClassificator(vocabulary_size=TaskA_datasets[\"A\"][\"vocabulary_size\"], embed_dimension=50, classes=2)\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
        "\n",
        "# Entrenamiento del modelo\n",
        "num_epochs = 5\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "model.to(device)\n",
        "\n",
        "for epoch in range(num_epochs):\n",
        "  model.train()\n",
        "  total_loss = 0\n",
        "\n",
        "  for batch_x, batch_y in train_dataloader:\n",
        "      batch_x, batch_y = batch_x.to(device), batch_y.to(device)\n",
        "\n",
        "      optimizer.zero_grad()\n",
        "      outputs = model(batch_x)\n",
        "      loss = criterion(outputs, batch_y)\n",
        "      loss.backward()\n",
        "      optimizer.step()\n",
        "\n",
        "      total_loss += loss.item()\n",
        "\n",
        "  print(f\"Epoch {epoch + 1}/{num_epochs}, Loss: {total_loss / len(train_dataloader)}\")\n",
        "\n",
        "# Evaluación del modelo\n",
        "model.eval()\n",
        "all_preds = []\n",
        "all_labels = []\n",
        "\n",
        "with torch.no_grad():\n",
        "  for batch_x, batch_y in test_dataloader:\n",
        "    batch_x, batch_y = batch_x.to(device), batch_y.to(device)\n",
        "\n",
        "    preds = model(batch_x)\n",
        "    _, pred_labels = torch.max(preds, 1)\n",
        "\n",
        "    all_preds.extend(pred_labels.cpu().numpy())\n",
        "    all_labels.extend(batch_y.cpu().numpy())\n",
        "\n",
        "# Calcula y muestra la accuracy\n",
        "accuracy = accuracy_score(all_labels, all_preds)\n",
        "print(f\"Accuracy on test set: {accuracy}\")"
      ],
      "metadata": {
        "id": "505ZECHS9Vd6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Define la función para convertir una lista de listas en tensores con padding dinámico:\n",
        "def dynamic_padding(batch):\n",
        "  # Ordena el batch por la longitud descendente:\n",
        "  sorted_batch = sorted(batch, key=lambda x: x[1], reverse=True)\n",
        "  sequences, lengths = zip(*sorted_batch)\n",
        "  padded_sequences = pad_sequence([torch.Tensor(seq) for seq in sequences], batch_first=True)\n",
        "  return padded_sequences, lengths"
      ],
      "metadata": {
        "id": "1xmVSgNRBnwV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def pad_data(X):\n",
        "    # Encuentra la longitud máxima de los ejemplos\n",
        "    max_length = max(len(example) for example in X)\n",
        "\n",
        "    # Inicializa un tensor con forma (num_examples, max_length, embed_dimension)\n",
        "    padded_X = torch.zeros((len(X), max_length, X[0].shape[1]))\n",
        "\n",
        "    # Rellena el tensor con los ejemplos\n",
        "    for i, example in enumerate(X):\n",
        "        length = len(example)\n",
        "        padded_X[i, :length, :] = torch.tensor(example)\n",
        "\n",
        "    return padded_X"
      ],
      "metadata": {
        "id": "ZM8hpdJI2gTp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import torch.optim as optim\n",
        "from torch.utils.data import TensorDataset, DataLoader\n",
        "\n",
        "# Obtén los datos y embeddings\n",
        "X, y, _ = get_data_and_embeddings(TaskA_datasets[\"A\"])\n",
        "max_length = max(len(example) for example in X)\n",
        "\n",
        "# Inicializa una lista para almacenar tensores de PyTorch\n",
        "padded_X = []\n",
        "\n",
        "# Convierte cada array de NumPy a un tensor de PyTorch y realiza el \"padding\" por lotes\n",
        "for example in X:\n",
        "    padded_example = np.zeros((max_length, len(example[0])), dtype=np.float32)\n",
        "    padded_example[:len(example)] = example\n",
        "    padded_X.append(torch.tensor(padded_example))\n",
        "\n",
        "# Divide los datos en conjuntos de entrenamiento y prueba\n",
        "X_train, X_test, y_train, y_test = train_test_split(padded_X, y, test_size=0.2, random_state=42)\n",
        "\n",
        "# Convierte los datos a tensores de PyTorch\n",
        "X_train_tensor = torch.tensor(X_train, dtype=torch.float32)\n",
        "y_train_tensor = torch.tensor(y_train, dtype=torch.float32)\n",
        "X_test_tensor = torch.tensor(X_test, dtype=torch.float32)\n",
        "y_test_tensor = torch.tensor(y_test, dtype=torch.float32)\n",
        "\n",
        "# Define el DataLoader para cargar los datos en lotes\n",
        "batch_size = 64\n",
        "train_data = TensorDataset(X_train_tensor, y_train_tensor)\n",
        "train_loader = DataLoader(train_data, batch_size=batch_size, shuffle=True)\n",
        "\n",
        "flat_X = np.concatenate([np.concatenate(example) for example in X])\n",
        "\n",
        "# Obtener palabras únicas utilizando operaciones de conjuntos\n",
        "unique_words_set = set(np.unique(flat_X))\n",
        "\n",
        "# Obtener el tamaño del vocabulario\n",
        "vocab_size = len(unique_words_set)\n",
        "\n",
        "# Hiperparámetros\n",
        "vocabulary_size = vocab_size\n",
        "embed_dimension = 50\n",
        "classes = 2\n",
        "learning_rate = 0.001\n",
        "epochs = 10\n",
        "\n",
        "# Instanciar el modelo\n",
        "model = AttentionalClassificator(vocabulary_size, embed_dimension, classes)\n",
        "\n",
        "# Optimizador\n",
        "optimizer = optim.Adam(model.parameters(), lr=learning_rate)\n",
        "criterion = nn.BCELoss()\n",
        "\n",
        "# Entrenamiento del modelo\n",
        "for epoch in range(epochs):\n",
        "    model.train()\n",
        "    running_loss = 0.0\n",
        "\n",
        "    for X_batch, y_batch in zip(X_train_tensor, y_train_tensor):\n",
        "        # Reiniciar los gradientes\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        # Pasar datos a través del modelo\n",
        "        outputs = model(X_batch)\n",
        "\n",
        "        # Calcular la pérdida\n",
        "        loss = criterion(outputs, y_batch.float())\n",
        "\n",
        "        # Retropropagación y optimización\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "\n",
        "        running_loss += loss.item()\n",
        "\n",
        "    # Imprimir la pérdida promedio en cada época\n",
        "    print(f'Epoch {epoch + 1}/{epochs}, Loss: {running_loss / len(X_train_tensor)}')\n",
        "\n",
        "# Validación del modelo en el conjunto de prueba\n",
        "model.eval()\n",
        "with torch.no_grad():\n",
        "    correct = 0\n",
        "    total = 0\n",
        "\n",
        "    for X_batch, y_batch in zip(X_test_tensor, y_test_tensor):\n",
        "        outputs = model(X_batch)\n",
        "        predicted = torch.round(outputs)\n",
        "        total += y_batch.size(0)\n",
        "        correct += (predicted == y_batch).sum().item()\n",
        "\n",
        "    accuracy = correct / total\n",
        "    print(f'Accuracy on test set: {accuracy}')"
      ],
      "metadata": {
        "id": "cTiby-Fn1rtu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# PARA TODOS LOS DATASETS:\n",
        "print(\"\\n════════════════════════════════════════════\")\n",
        "print(\"\\nSELF-ATTENTION ENCODER CLASSIFICATOR MODEL\")\n",
        "print(\"\\n════════════════════════════════════════════\")\n",
        "for dataset_name, dataset_info in TaskA_datasets.items():\n",
        "  print(f\"\\nDATASET {dataset_name}:\")\n",
        "\n",
        "  # Obtiene datos y embeddings:\n",
        "  X, y, unique_words = get_data_and_embeddings(dataset_info)\n",
        "\n",
        "  # Instancia el modelo:\n",
        "  model = AttentionalClassificator(vocabulary_size=len(unique_words), embed_dimension=50, classes=2)\n",
        "\n",
        "  # Entrena y evalúa el modelo con k-fold cross validation:\n",
        "  avg_accuracy, avg_macro_f1, avg_micro_f1 = train_and_evaluate_model(model, X, y, epochs=1000, kfolds=5)\n",
        "\n",
        "  # Imprime resultados:\n",
        "  print(f\" * Promedio Accuracy: {avg_accuracy}\")\n",
        "  print(f\" * Promedio Macro F1: {avg_macro_f1}\")\n",
        "  print(f\" * Promedio Micro F1: {avg_micro_f1}\")\n",
        "\n",
        "print(\"\\n════════════════════════════════════════════\")"
      ],
      "metadata": {
        "id": "ZAezE9HUltSY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import StratifiedKFold\n",
        "from sklearn.metrics import accuracy_score, f1_score\n",
        "from sklearn.preprocessing import LabelEncoder, OneHotEncoder\n",
        "from torch.utils.data import DataLoader\n",
        "\n",
        "# Función para entrenar y evaluar con K-Folds:\n",
        "def train_and_evaluate_model(model, X, y, epochs=100, kfolds=5):\n",
        "  skf = StratifiedKFold(n_splits=kfolds, shuffle=True, random_state=11)\n",
        "\n",
        "  accuracies = []\n",
        "  macro_f1_scores = []\n",
        "  micro_f1_scores = []\n",
        "\n",
        "  # One-Hot Encoding para etiquetas:\n",
        "  label_encoder = LabelEncoder()\n",
        "  onehot_encoder = OneHotEncoder(sparse=False)\n",
        "  y_encoded = label_encoder.fit_transform(y)\n",
        "  y_onehot = onehot_encoder.fit_transform(y_encoded.reshape(-1, 1))\n",
        "\n",
        "  for train_index, _ in skf.split(X, y):\n",
        "    X_train, y_train = X[train_index], y_onehot[train_index]\n",
        "\n",
        "    # Tamaño máximo de índices del entrenamiento actual:\n",
        "    max_index = int(np.max(np.concatenate([np.max(seq) for seq in X_train], axis=None)))\n",
        "    max_len = int(np.max([len(seq) for seq in X_train]))\n",
        "    model.positional = PositionalEncoding(model.embeddings.embedding_dim, max_len=max_len)\n",
        "\n",
        "\n",
        "    # Verifica y actualiza el tamaño del vocabulario en la capa de embeddings:\n",
        "    if max_index >= model.embeddings.num_embeddings:\n",
        "      new_vocab_size = max_index + 1\n",
        "      new_embeddings = nn.Embedding(new_vocab_size, model.embeddings.embedding_dim)\n",
        "      with torch.no_grad():\n",
        "        new_embeddings.weight[:model.embeddings.num_embeddings, :] = model.embeddings.weight\n",
        "      model.embeddings = new_embeddings\n",
        "\n",
        "    # Convierte x, y a tensores:\n",
        "    dataloader = DataLoader(list(zip(X_train, [len(seq) for seq in X_train])), batch_size=20, collate_fn=dynamic_padding, shuffle=True)\n",
        "    y_train_tensor = torch.tensor(y_train, dtype=torch.float32)\n",
        "\n",
        "    # Define la pérdida y el optimizador:\n",
        "    criterion = nn.BCEWithLogitsLoss()\n",
        "    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n",
        "\n",
        "    # Entrenamiento:\n",
        "    for epoch in tqdm(range(epochs)):\n",
        "      for batch, lengths in dataloader:\n",
        "        X_batch = batch\n",
        "\n",
        "        # Hace el padding dinámico:\n",
        "        X_packed = pack_padded_sequence(X_batch, lengths, batch_first=True, enforce_sorted=False)\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        # Desempaqueta X_packed antes de pasar a la capa de embedding:\n",
        "        X_unpacked, _ = pad_packed_sequence(X_packed, batch_first=True)\n",
        "        X_unpacked = X_unpacked.long()\n",
        "\n",
        "        # Resto del entrenamiento:\n",
        "        output = model(X_unpacked)\n",
        "        loss = criterion(output, y_train_tensor)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "\n",
        "      # Evaluación:\n",
        "      model.eval()\n",
        "      with torch.no_grad():\n",
        "        padded_sequences, _ = pad_packed_sequence(X_packed, batch_first=True)\n",
        "        y_pred = model(padded_sequences).argmax(dim=1).numpy()\n",
        "      accuracies.append(accuracy_score(y_train, y_pred))\n",
        "      macro_f1_scores.append(f1_score(y_train, y_pred, average=\"macro\"))\n",
        "      micro_f1_scores.append(f1_score(y_train, y_pred, average=\"micro\"))\n",
        "\n",
        "  # Calcula promedio de métricas:\n",
        "  avg_accuracy = np.mean(accuracies)\n",
        "  avg_macro_f1 = np.mean(macro_f1_scores)\n",
        "  avg_micro_f1 = np.mean(micro_f1_scores)\n",
        "\n",
        "  return avg_accuracy, avg_macro_f1, avg_micro_f1"
      ],
      "metadata": {
        "id": "rFBP78j0hvbj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.svm import SVC\n",
        "from sklearn.metrics import accuracy_score\n",
        "\n",
        "# Obtén los datos de entrenamiento y embeddings para el conjunto A\n",
        "X_A, y_A, _ = get_data_and_embeddings(TaskA_datasets[\"A\"])\n",
        "\n",
        "# Divide los datos en conjuntos de entrenamiento y prueba\n",
        "X_train_A, X_test_A, y_train_A, y_test_A = train_test_split(X_A, y_A, test_size=0.2, random_state=42)\n",
        "\n",
        "# Entrenar el modelo utilizando lotes\n",
        "model_A = SVC()\n",
        "\n",
        "batch_size = 100\n",
        "for i in range(0, len(X_train_A), batch_size):\n",
        "    X_batch = [np.mean(np.concatenate(x_i), axis=0).reshape(1, -1) for x_i in X_train_A[i:i+batch_size]]\n",
        "    y_batch = y_train_A[i:i+batch_size]\n",
        "    model_A.fit(np.concatenate(X_batch), y_batch)"
      ],
      "metadata": {
        "id": "f-8S8wOft9h-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Realizar predicciones en el conjunto de prueba\n",
        "X_test_flat_A = []\n",
        "\n",
        "for x_i in X_test_A:\n",
        "  # Promedia los vectores de palabras y da la forma adecuada\n",
        "  averaged_vector = np.mean(np.concatenate(x_i), axis=0).reshape(1, -1)\n",
        "  X_test_flat_A.append(averaged_vector)\n",
        "\n",
        "# Concatenar la lista de vectores en un array bidimensional\n",
        "X_test_flat_A = np.concatenate(X_test_flat_A)\n",
        "\n",
        "# Realizar predicciones en el conjunto de prueba\n",
        "predictions_A = model_A.predict(X_test_flat_A)\n",
        "\n",
        "# Evaluar la precisión del modelo\n",
        "accuracy_A = accuracy_score(y_test_A, predictions_A)\n",
        "print(f\"Accuracy for Task A: {accuracy_A}\")\n"
      ],
      "metadata": {
        "id": "SbfvE-sTwjLB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"X_test_A.shape:\", X_test_A.shape)\n",
        "print(\"y_test_A.shape:\", y_test_A)\n",
        "print(\"y_test_A[0]:\", y_test_A[0])\n",
        "print(\"y_test_A[80]:\", y_test_A[80])"
      ],
      "metadata": {
        "id": "YO2mZc-5xMhP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## MÁS EJEMPLOS FALLIDOS"
      ],
      "metadata": {
        "id": "-LQvpRc8yIgc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "from torch.utils.data import DataLoader, TensorDataset\n",
        "from sklearn.metrics import accuracy_score\n",
        "from torch.nn.utils.rnn import pad_sequence, pack_padded_sequence\n",
        "from sklearn.model_selection import StratifiedKFold\n",
        "from sklearn.metrics import accuracy_score, f1_score\n",
        "from sklearn.preprocessing import LabelEncoder, OneHotEncoder"
      ],
      "metadata": {
        "id": "HTKMu1dUmvIT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Define la función para convertir una lista de listas en tensores con padding dinámico:\n",
        "def dynamic_padding(batch):\n",
        "  # Ordena el batch por la longitud descendente:\n",
        "  sorted_batch = sorted(batch, key=lambda x: x[1], reverse=True)\n",
        "  sequences, lengths = zip(*sorted_batch)\n",
        "  print(f\"\\nLengths of sequences: {lengths}\")\n",
        "\n",
        "  # Convierte las secuencias a tensores de tipo entero:\n",
        "  sequences = [torch.LongTensor(np.array(seq).astype(np.int64)) for seq in sequences]\n",
        "\n",
        "  # Utiliza pad_sequence directamente:\n",
        "  padded_sequences = pad_sequence(sequences, batch_first=True)\n",
        "\n",
        "  # Asegura que las longitudes reflejen las longitudes reales después del padding:\n",
        "  lengths = [len(seq) for seq in sequences]\n",
        "\n",
        "  return padded_sequences, lengths"
      ],
      "metadata": {
        "id": "ALZzu1-tbbp7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Función para entrenar y evaluar con K-Folds:\n",
        "def train_and_evaluate_model(model, X, y, epochs=100, kfolds=5):\n",
        "  skf = StratifiedKFold(n_splits=kfolds, shuffle=True, random_state=11)\n",
        "\n",
        "  accuracies = []\n",
        "  macro_f1_scores = []\n",
        "  micro_f1_scores = []\n",
        "\n",
        "  # One-Hot Encoding para etiquetas:\n",
        "  label_encoder = LabelEncoder()\n",
        "  onehot_encoder = OneHotEncoder(sparse=False)\n",
        "  y_encoded = label_encoder.fit_transform(y)\n",
        "  y_onehot = onehot_encoder.fit_transform(y_encoded.reshape(-1, 1))\n",
        "\n",
        "  for train_index, _ in skf.split(X, y):\n",
        "    X_train, y_train = X[train_index], y_onehot[train_index]\n",
        "\n",
        "    # Convierte x, y a tensores:\n",
        "    dataloader = DataLoader(list(zip(X_train, [len(seq) for seq in X_train])), batch_size=1, collate_fn=dynamic_padding, shuffle=True)\n",
        "    y_train_tensor = torch.tensor(y_train, dtype=torch.float32)\n",
        "\n",
        "    # Define la pérdida y el optimizador:\n",
        "    criterion = nn.BCEWithLogitsLoss()\n",
        "    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n",
        "\n",
        "    # Entrenamiento:\n",
        "    for epoch in tqdm(range(epochs)):\n",
        "      for batch, lengths in dataloader:\n",
        "        X_batch = batch\n",
        "\n",
        "        # Hace el padding dinámico:\n",
        "        X_packed = pack_padded_sequence(X_batch, lengths, batch_first=True, enforce_sorted=False)\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        # Desempaqueta X_packed antes de pasar a la capa de embedding:\n",
        "        X_unpacked, _ = pad_packed_sequence(X_packed, batch_first=True)\n",
        "        print(f\"Shape of X_unpacked: {X_unpacked.shape}\")\n",
        "        X_unpacked = X_unpacked.long()\n",
        "\n",
        "        # Resto del entrenamiento:\n",
        "        output = model(X_unpacked)\n",
        "        loss = criterion(output, y_train_tensor)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "\n",
        "      # Evaluación:\n",
        "      model.eval()\n",
        "      with torch.no_grad():\n",
        "        padded_sequences, _ = pad_packed_sequence(X_packed, batch_first=True)\n",
        "        print(f\"Shape of padded_sequences: {padded_sequences.shape}\")\n",
        "        y_pred = model(padded_sequences).argmax(dim=1).numpy()\n",
        "      accuracies.append(accuracy_score(y_train, y_pred))\n",
        "      macro_f1_scores.append(f1_score(y_train, y_pred, average=\"macro\"))\n",
        "      micro_f1_scores.append(f1_score(y_train, y_pred, average=\"micro\"))\n",
        "    print()\n",
        "\n",
        "  # Calcula promedio de métricas:\n",
        "  avg_accuracy = np.mean(accuracies)\n",
        "  avg_macro_f1 = np.mean(macro_f1_scores)\n",
        "  avg_micro_f1 = np.mean(micro_f1_scores)\n",
        "\n",
        "  return avg_accuracy, avg_macro_f1, avg_micro_f1"
      ],
      "metadata": {
        "id": "2jl7Z3xep_Ns"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# PARA TODOS LOS DATASETS:\n",
        "print(\"\\n════════════════════════════════════════════\")\n",
        "print(\"\\nSELF-ATTENTION ENCODER CLASSIFICATOR MODEL\")\n",
        "print(\"\\n════════════════════════════════════════════\")\n",
        "for dataset_name, dataset_info in TaskA_datasets.items():\n",
        "  print(f\"\\nDATASET {dataset_name}:\")\n",
        "\n",
        "  # Obtiene datos y embeddings:\n",
        "  # \"size\" es un porcentaje; si es 1, se usará el 100% del dataset, si es 0.5, el 50%, etc:\n",
        "  X, y, unique_words = get_data_and_embeddings(dataset_info, size=0.0001)\n",
        "\n",
        "\n",
        "  print(f\"Shape of X: {X.shape}\")\n",
        "  print(f\"Shape of y: {y.shape}\")\n",
        "  print(f\"Number of unique words: {unique_words}\")\n",
        "\n",
        "  # Instancia el modelo:\n",
        "  model = AttentionalClassificator(vocabulary_size=unique_words, embed_dimension=50, classes=2)\n",
        "\n",
        "  # Entrena y evalúa el modelo con k-fold cross validation:\n",
        "  avg_accuracy, avg_macro_f1, avg_micro_f1 = train_and_evaluate_model(model, X, y, epochs=1000, kfolds=5)\n",
        "\n",
        "  # Imprime resultados:\n",
        "  print(f\" * Promedio Accuracy: {avg_accuracy}\")\n",
        "  print(f\" * Promedio Macro F1: {avg_macro_f1}\")\n",
        "  print(f\" * Promedio Micro F1: {avg_micro_f1}\")\n",
        "\n",
        "print(\"\\n════════════════════════════════════════════\")"
      ],
      "metadata": {
        "id": "7m0NHIpRbb_V"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "X, y, unique_words = get_data_and_embeddings(TaskA_datasets[\"A\"], size=0.0001)"
      ],
      "metadata": {
        "id": "-T9uBGqCL7yc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Cantidad de ejemplos de entrenamiento:\n",
        "len(X)"
      ],
      "metadata": {
        "id": "EZvY0VyqPmUF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Longitud de palabras del primer ejemplo:\n",
        "len(X[0])"
      ],
      "metadata": {
        "id": "qptqOP3MP281"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Longitud de la primer palabra del primer ejemplo:\n",
        "len(X[0][0])"
      ],
      "metadata": {
        "id": "NalFn5vlP92v"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Secuencia más grande:\n",
        "max_sequence_length"
      ],
      "metadata": {
        "id": "4XFlOXDQiTBY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "from torch.utils.data import DataLoader\n",
        "from tqdm import tqdm\n",
        "\n",
        "def train_model(model, train_loader, criterion, optimizer, device):\n",
        "    model.train()\n",
        "    total_loss = 0.0\n",
        "\n",
        "    for inputs, labels in tqdm(train_loader, desc=\"Training\", unit=\"batch\"):\n",
        "        inputs, labels = inputs.squeeze(0).to(device), labels.to(device)\n",
        "\n",
        "        print(f\"\\nInput shape (before training): {inputs.shape}\")\n",
        "\n",
        "        print(\"Haciendo optimizer.zero_grad()\")\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        print(\"Haciendo outputs = model(inputs)\")\n",
        "        outputs = model(inputs)\n",
        "        print(\"Haciendo loss = criterion(outputs, labels)\")\n",
        "        loss = criterion(outputs, labels)\n",
        "        print(\"Haciendo loss.backward()\")\n",
        "        loss.backward()\n",
        "        print(\"Haciendo optimizer.step()\")\n",
        "        optimizer.step()\n",
        "\n",
        "        total_loss += loss.item()\n",
        "\n",
        "        # Añadir este print statement para verificar las dimensiones\n",
        "        print(f\"Input shape (during training): {inputs.shape}\")\n",
        "\n",
        "    return total_loss / len(train_loader)\n",
        "\n",
        "def evaluate_model(model, val_loader, criterion, device):\n",
        "    model.eval()\n",
        "    total_loss = 0.0\n",
        "    correct_predictions = 0\n",
        "    total_samples = 0\n",
        "\n",
        "    with torch.no_grad():\n",
        "        for inputs, labels in tqdm(val_loader, desc=\"Evaluating\", unit=\"batch\"):\n",
        "            inputs, labels = inputs.to(device), labels.to(device)\n",
        "\n",
        "            outputs = model(inputs)\n",
        "            loss = criterion(outputs, labels)\n",
        "            total_loss += loss.item()\n",
        "\n",
        "            _, predicted = torch.max(outputs, 1)\n",
        "            correct_predictions += (predicted == labels).sum().item()\n",
        "            total_samples += labels.size(0)\n",
        "\n",
        "    average_loss = total_loss / len(val_loader)\n",
        "    accuracy = correct_predictions / total_samples\n",
        "\n",
        "    return average_loss, accuracy"
      ],
      "metadata": {
        "id": "92D6APK1RSR7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "os.environ['CUDA_LAUNCH_BLOCKING'] = '1'\n",
        "# Verifica si CUDA está disponible\n",
        "print(\"CUDA available:\", torch.cuda.is_available())\n",
        "\n",
        "# Imprime información sobre el GPU actual\n",
        "if torch.cuda.is_available():\n",
        "    print(\"GPU:\", torch.cuda.get_device_name(0))\n",
        "\n",
        "print(\"Versión de PyTorch:\", torch.__version__)\n",
        "print(\"Versión de CUDA:\", torch.version.cuda)\n",
        "\n",
        "torch.autograd.set_detect_anomaly(True)"
      ],
      "metadata": {
        "id": "oug4FF_dfGBF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from torch.nn.utils.rnn import pad_sequence\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "# Dividir el conjunto de datos en entrenamiento y prueba\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=11)\n",
        "\n",
        "# Obtener la longitud máxima de las secuencias\n",
        "max_sequence_length = max(len(seq) for seq in X_train + X_test)\n",
        "\n",
        "# Asegurarse de que todas las secuencias tengan la misma longitud (ajustar según sea necesario)\n",
        "X_train = [seq[:max_sequence_length] for seq in X_train]\n",
        "X_test = [seq[:max_sequence_length] for seq in X_test]\n",
        "\n",
        "# Rellenar las secuencias más cortas con ceros\n",
        "X_train_padded = pad_sequence([torch.Tensor(seq) for seq in X_train], batch_first=True)\n",
        "X_test_padded = pad_sequence([torch.Tensor(seq) for seq in X_test], batch_first=True)\n",
        "\n",
        "# Verificar las dimensiones después de rellenar\n",
        "print(\"X_train_padded shape:\", X_train_padded.shape)\n",
        "print(\"X_test_padded shape:\", X_test_padded.shape)\n",
        "print(\"X_train_padded[0] length:\", len(X_train_padded[0]))\n",
        "print(\"X_test_padded[0] length:\", len(X_test_padded[0]))\n",
        "# Antes de la creación del conjunto de datos\n",
        "print(f\"X_train_padded shape before creating dataset: {X_train_padded.shape}\")\n",
        "\n",
        "# Verificar que los índices en las secuencias estén dentro del rango del vocabulario\n",
        "# Imprimir las longitudes de las secuencias en el conjunto de entrenamiento\n",
        "print(\"Longitudes de secuencias en entrenamiento:\")\n",
        "for i, seq in enumerate(X_train_padded):\n",
        "    print(f\"Secuencia {i + 1}: {len(seq)}\")\n",
        "\n",
        "# Imprimir las longitudes de las secuencias en el conjunto de prueba\n",
        "print(\"\\nLongitudes de secuencias en prueba:\")\n",
        "for i, seq in enumerate(X_test_padded):\n",
        "    print(f\"Secuencia {i + 1}: {len(seq)}\")\n",
        "\n",
        "# Crear TensorDataset y DataLoader\n",
        "train_dataset = torch.utils.data.TensorDataset(X_train_padded.long(), torch.LongTensor(y_train))\n",
        "test_dataset = torch.utils.data.TensorDataset(X_test_padded.long(), torch.LongTensor(y_test))\n",
        "\n",
        "train_loader = DataLoader(train_dataset, batch_size=1, shuffle=True)\n",
        "test_loader = DataLoader(test_dataset, batch_size=1, shuffle=False)\n",
        "\n",
        "# Definir modelo, optimizador y criterio\n",
        "model = AttentionalClassificator(unique_words, embed_dimension=50, classes=2)\n",
        "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "\n",
        "# Entrenamiento\n",
        "num_epochs = 10\n",
        "device = torch.device(\"cpu\")\n",
        "model.to(device)\n",
        "\n",
        "for epoch in range(num_epochs):\n",
        "    train_loss = train_model(model, train_loader, criterion, optimizer, device)\n",
        "    print(f\"Epoch {epoch + 1}/{num_epochs}, Training Loss: {train_loss:.4f}\")\n",
        "\n",
        "# Evaluación\n",
        "test_loss, test_accuracy = evaluate_model(model, test_loader, criterion, device)\n",
        "print(f\"Test Loss: {test_loss:.4f}, Test Accuracy: {test_accuracy * 100:.2f}%\")"
      ],
      "metadata": {
        "id": "xPWihFnPQ86G"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## X"
      ],
      "metadata": {
        "id": "R8h2nmsIH9By"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Codificar etiquetas\n",
        "label_encoder = LabelEncoder()\n",
        "y_encoded = label_encoder.fit_transform(y)\n",
        "\n",
        "# Dividir datos en conjuntos de entrenamiento y prueba\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y_encoded, test_size=0.3, random_state=11)\n",
        "\n",
        "# Obtener la longitud máxima de las secuencias\n",
        "max_sequence_length = max(len(seq) for seq in X_train + X_test)\n",
        "\n",
        "# Obtener índices de palabras a partir de word2Index:\n",
        "X_train_indices = [[word2Index[word] for word in texto.split()] for texto in X_train[\"text\"]]\n",
        "X_test_indices = [[word2Index[word] for word in texto.split()] for texto in X_test[\"text\"]]\n",
        "\n",
        "# Asegurarse de que todas las secuencias tengan la misma longitud (ajustar según sea necesario)\n",
        "X_train = [seq[:max_sequence_length] for seq in X_train_indices]\n",
        "X_test = [seq[:max_sequence_length] for seq in X_test_indices]\n",
        "\n",
        "# Rellenar las secuencias más cortas con ceros\n",
        "X_train_padded = pad_sequence([torch.Tensor(seq) for seq in X_train], batch_first=True)\n",
        "X_test_padded = pad_sequence([torch.Tensor(seq) for seq in X_test], batch_first=True)\n",
        "\n",
        "# Verificar las dimensiones después de rellenar\n",
        "print(\"max_sequence_length:\", max_sequence_length)\n",
        "print(\"X_train_padded shape:\", X_train_padded.shape)\n",
        "print(\"X_test_padded shape:\", X_test_padded.shape)\n",
        "print(\"X_train_padded[0] length:\", len(X_train_padded[0]))\n",
        "print(\"X_test_padded[0] length:\", len(X_test_padded[0]))\n",
        "# Antes de la creación del conjunto de datos\n",
        "print(f\"X_train_padded shape before creating dataset: {X_train_padded.shape}\")\n",
        "\n",
        "# Verificar que los índices en las secuencias estén dentro del rango del vocabulario\n",
        "# Imprimir las longitudes de las secuencias en el conjunto de entrenamiento\n",
        "print(\"Longitudes de secuencias en entrenamiento:\")\n",
        "for i, seq in enumerate(X_train_padded):\n",
        "    print(f\"Secuencia {i + 1}: {len(seq)}\")\n",
        "\n",
        "# Imprimir las longitudes de las secuencias en el conjunto de prueba\n",
        "print(\"\\nLongitudes de secuencias en prueba:\")\n",
        "for i, seq in enumerate(X_test_padded):\n",
        "    print(f\"Secuencia {i + 1}: {len(seq)}\")\n",
        "\n",
        "# Crear TensorDataset y DataLoader\n",
        "train_dataset = torch.utils.data.TensorDataset(X_train_padded.long(), torch.tensor(y_train))\n",
        "test_dataset = torch.utils.data.TensorDataset(X_test_padded.long(), torch.tensor(y_test))\n",
        "\n",
        "train_loader = DataLoader(train_dataset, batch_size=1, shuffle=True)\n",
        "test_loader = DataLoader(test_dataset, batch_size=1, shuffle=False)\n",
        "\n",
        "# Inicializar el modelo:\n",
        "# device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "device = \"cpu\"\n",
        "model = SelfAttentionClassifier(embedding_dim=50, hidden_dim=64, num_heads=5, output_dim=len(set(y)), dropout=0.1).to(device)\n",
        "\n",
        "# Definir la función de pérdida y el optimizador\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "optimizer = optim.Adam(model.parameters(), lr=0.001)\n",
        "\n",
        "# Entrenar el modelo\n",
        "epochs = 10\n",
        "for epoch in range(epochs):\n",
        "    model.train()\n",
        "    for batch_X, batch_y in train_loader:\n",
        "        optimizer.zero_grad()\n",
        "        outputs = model(batch_X.to(device))\n",
        "        loss = criterion(outputs, batch_y.to(device))\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "\n",
        "# Evaluar el modelo en el conjunto de prueba\n",
        "model.eval()\n",
        "with torch.no_grad():\n",
        "    predictions = []\n",
        "    true_labels = []\n",
        "    for batch_X, batch_y in test_loader:\n",
        "        outputs = model(batch_X.cuda())\n",
        "        _, predicted = torch.max(outputs, 1)\n",
        "        predictions.extend(predicted.cpu().numpy())\n",
        "        true_labels.extend(batch_y.cpu().numpy())"
      ],
      "metadata": {
        "id": "9Aty2SJMIAAq"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}