data_vectorizer.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "data_vectorizer.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyOqMBpZ1jdAFI1i/E9VvUZo"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Introduction\n",
        "We use this notebook to vectorize the text data. \n",
        "\n",
        "As mentioned in [simpletransformers](https://simpletransformers.ai/docs/text-rep-model/):\n",
        "> \"The RepresentationModel class is used for generating (contextual) word or sentence embeddings from a list of text sentences, You can then feed these vectors to any model or downstream task.\"\n",
        "\n",
        "After we vectorize the data, it will be used for data augmentation or any other task phases in other notebooks."
      ],
      "metadata": {
        "id": "fOsMASqd7fYV"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mL408psinDvY"
      },
      "source": [
        "!pip install simpletransformers\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "from simpletransformers.language_representation import RepresentationModel\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4EH2tMAenLoQ"
      },
      "source": [
        "train_data = pd.read_csv(\"/content/drive/MyDrive/TUBITAK_TASK1/data/stweet/train.csv\")\n",
        "test_data = pd.read_csv('/content/drive/MyDrive/TUBITAK_TASK1/data/stweet/test.csv')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iUC2fIWeuxY8",
        "outputId": "9257f2bc-c8ac-4e25-e84b-a7d655135aca"
      },
      "source": [
        "print(train_data.shape)\n",
        "print(test_data.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(1046343, 2)\n",
            "(478955, 2)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kPMSm0BkrA9o"
      },
      "source": [
        "## Both test and train set must be vectorized by BERT \n",
        "train = train_data[\"text\"]\n",
        "test = test_data[\"text\"]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "t3kDq0bYxKAL",
        "outputId": "68dfcca2-eb43-436d-c6c0-1eded5f98904"
      },
      "source": [
        "print(train.shape)\n",
        "print(test.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(1046343,)\n",
            "(478955,)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "_____\n",
        "I did not vectorize every single sentence for our case due to lack of space and time. Creating a million sentences vector is taking long time and needs to 10GB+ disk space. Therefore, we decrease the size of dataset while vectorizing. In this notebook 150k sentences was found the max available choice for RAM in Colab. However, you can try for larger vectors if you have the hardware.\n",
        "\n",
        "We vectorize both train and test set seperately and \"bert-base-uncased\" will be using in most of our cases.\n",
        "\n",
        "I used encoded_text phrase for vectorized data rest of the study, but you need to basically see as it becomes numerical text in 768 length through BERT.\n",
        "_____"
      ],
      "metadata": {
        "id": "S5c5sNmx86e3"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hApddQ9NrSeU"
      },
      "source": [
        "# use this block if new bert vectors needed\n",
        "# without CUDA=true it is nearly imposible to vectorize\n",
        "\n",
        "# TRAIN SET\n",
        "\n",
        "sentences = train[:150000]\n",
        "model = RepresentationModel(model_type=\"bert\", model_name='bert-base-uncased', use_cuda=True)\n",
        "word_vectors_train = model.encode_sentences(sentences, combine_strategy=\"mean\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "siAe72Ns-08t"
      },
      "source": [
        "word_vectors_train.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dsWKth-1rjFq"
      },
      "source": [
        "# Saving the word_vectors to drive cuz it takes long hours u know\n",
        "\n",
        "df_train = pd.DataFrame(word_vectors_train)\n",
        "df_train.to_csv('/content/drive/MyDrive/TUBITAK_TASK1/encoded_texts/stweet/encoded_stweet150000_train.csv')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gwcgegO1rVhy",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "667392a7-d007-4bb2-8d21-c9fb4cf796f2"
      },
      "source": [
        "# use this block if new bert vectors needed\n",
        "# TEST SET\n",
        "\n",
        "sentences = test[:20000]\n",
        "model = RepresentationModel(model_type=\"bert\", model_name='bert-base-uncased', use_cuda=True)\n",
        "word_vectors_test = model.encode_sentences(sentences, combine_strategy= \"mean\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForTextRepresentation: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias']\n",
            "- This IS expected if you are initializing BertForTextRepresentation from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertForTextRepresentation from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4-1IxZ1NrzoN"
      },
      "source": [
        "# Saving the word_vectors to drive cuz it takes long hours u know\n",
        "\n",
        "df_test = pd.DataFrame(word_vectors_test)\n",
        "df_test.to_csv('/content/drive/MyDrive/TUBITAK_TASK1/encoded_texts/stweet/encoded_stweet20000_test.csv')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e6QIkxfMsAOR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ff4d0054-f301-41b0-b74c-7c7294d9cb88"
      },
      "source": [
        "print(\"X_train shape:\", word_vectors_train.shape)\n",
        "print(\"X_test shape:\", word_vectors_test.shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "X_train shape: (150000, 768)\n",
            "X_test shape: (20000, 768)\n"
          ]
        }
      ]
    }
  ]
}