SciAbstractClassifier.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "SciAbstractClassifier.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyOA1JSCoBCmnId6FhDzvfHy",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/lasupernova/AI_Nim_Game/blob/master/SciAbstractClassifier.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5tz0z1GzcOJZ"
      },
      "source": [
        "An NLP model that makes reading scientific abstract easier by separating an abstract into 5 parts.\n",
        "\n",
        " - Background\n",
        " - Objectives\n",
        " - Methods\n",
        " - Results\n",
        " - Conclusion\n",
        "\n",
        "The model aims to replicate and modify (improved) the models built in the following paper:\n",
        "https://arxiv.org/abs/1710.06071"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0lQ6gTRsX6O4",
        "outputId": "8b55be2a-aeea-4a64-d7b0-f54824f12f5a"
      },
      "source": [
        "#check access to GPU\n",
        "!nvidia-smi -L"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "GPU 0: Tesla K80 (UUID: GPU-46674db3-6d55-acab-3481-f7063e3d91e6)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LzCDoIbgctl_"
      },
      "source": [
        "The PubMed 200k RCT dataset (https://github.com/Franck-Dernoncourt/pubmed-rct)  will be used for training the model, based on the following paper:\n",
        "\n",
        "https://arxiv.org/abs/1710.06071"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hMJPjCVYcost",
        "outputId": "60e9af4b-d3b8-4ef7-b15b-ef86258123cc"
      },
      "source": [
        "#download data\n",
        "!git clone https://github.com/Franck-Dernoncourt/pubmed-rct\n",
        "!ls pubmed-rct"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'pubmed-rct'...\n",
            "remote: Enumerating objects: 33, done.\u001b[K\n",
            "remote: Counting objects: 100% (3/3), done.\u001b[K\n",
            "remote: Compressing objects: 100% (3/3), done.\u001b[K\n",
            "remote: Total 33 (delta 0), reused 0 (delta 0), pack-reused 30\u001b[K\n",
            "Unpacking objects: 100% (33/33), done.\n",
            "PubMed_200k_RCT\n",
            "PubMed_200k_RCT_numbers_replaced_with_at_sign\n",
            "PubMed_20k_RCT\n",
            "PubMed_20k_RCT_numbers_replaced_with_at_sign\n",
            "README.md\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X8e7q_DcJ-dU",
        "outputId": "a7fdf8a5-813a-492b-841c-6825dc49383c"
      },
      "source": [
        "#download helper function to use throughout script\n",
        "!wget https://raw.githubusercontent.com/lasupernova/Tensorflow_apps/master/helpers.py"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2021-10-12 22:18:44--  https://raw.githubusercontent.com/lasupernova/Tensorflow_apps/master/helpers.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.110.133, 185.199.111.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 10252 (10K) [text/plain]\n",
            "Saving to: ‘helpers.py’\n",
            "\n",
            "\rhelpers.py            0%[                    ]       0  --.-KB/s               \rhelpers.py          100%[===================>]  10.01K  --.-KB/s    in 0s      \n",
            "\n",
            "2021-10-12 22:18:44 (57.9 MB/s) - ‘helpers.py’ saved [10252/10252]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wyAO8lqHe7un"
      },
      "source": [
        "#imports\n",
        "import os\n",
        "import statistics\n",
        "import random\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.preprocessing import OneHotEncoder, LabelEncoder\n",
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "from sklearn.pipeline import Pipeline\n",
        "import tensorflow_hub as hub\n",
        "from helpers import calculate_results\n",
        "import tensorflow as tf"
      ],
      "execution_count": 78,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nz8MweW6d2OY"
      },
      "source": [
        "## Model using 10% of data set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xd65P5B9d11H",
        "outputId": "80497880-40da-4d17-b05f-8c46a651b61d"
      },
      "source": [
        "#check pubmed_20k_rct dataset files\n",
        "!ls pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "dev.txt  test.txt  train.txt\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SRIjQCSWeXZu"
      },
      "source": [
        "NOTE: <br>\n",
        "train.txt - train dataset<br>\n",
        "test.txt - test dataset<br>\n",
        "dev.txt - validation dataset<br>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dEPzjAcTddvQ"
      },
      "source": [
        "DATA_DIR = \"/content/pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign\""
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kt34l0Zbe3HK",
        "outputId": "4efd4dc0-0d69-413f-da1f-3f199e4a7f37"
      },
      "source": [
        "#get list of filenames in dir\n",
        "f_names = [f\"{DATA_DIR}{os.sep}{f_name}\" for f_name in os.listdir(DATA_DIR)]\n",
        "f_names"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['/content/pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/test.txt',\n",
              " '/content/pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/train.txt',\n",
              " '/content/pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/dev.txt']"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7KThNA05hT-k"
      },
      "source": [
        "### Pre-process data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mBVCZjJdfNxi"
      },
      "source": [
        "#read data into variables\n",
        "def import_data_from_txt(f_name):\n",
        "  \"\"\"\n",
        "  Read data contained in specified file and save data (text) to variable to.\n",
        "\n",
        "  PARAMS:\n",
        "    f_name(str) - path to file\n",
        "\n",
        "  RETURNS:\n",
        "    data(lst) - list with text data save as entries\n",
        "  \"\"\"\n",
        "  extension = f_name.split(\".\")[-1]\n",
        "  if extension != \"txt\":\n",
        "    print(\"ERROR! Extension of type .txt required!\")\n",
        "  with open(f_name, \"r\") as f:\n",
        "    return f.readlines()"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y7AKWuskiaO_"
      },
      "source": [
        "# for file in f_names:\n",
        "#   name = file.split(\"/\")[-1].split(\".\")[0]\n",
        "#   name = "
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nWCKyfyLjI5c"
      },
      "source": [
        "train_data = import_data_from_txt('/content/pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/train.txt')\n",
        "test_data = import_data_from_txt('/content/pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/test.txt')\n",
        "val_data = import_data_from_txt('/content/pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/dev.txt')"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CNvt1THikEU0",
        "outputId": "28c0c44c-79ae-46e2-a4a0-3ca6c6d42057"
      },
      "source": [
        "train_data[:20]"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['###24293578\\n',\n",
              " 'OBJECTIVE\\tTo investigate the efficacy of @ weeks of daily low-dose oral prednisolone in improving pain , mobility , and systemic low-grade inflammation in the short term and whether the effect would be sustained at @ weeks in older adults with moderate to severe knee osteoarthritis ( OA ) .\\n',\n",
              " 'METHODS\\tA total of @ patients with primary knee OA were randomized @:@ ; @ received @ mg/day of prednisolone and @ received placebo for @ weeks .\\n',\n",
              " 'METHODS\\tOutcome measures included pain reduction and improvement in function scores and systemic inflammation markers .\\n',\n",
              " 'METHODS\\tPain was assessed using the visual analog pain scale ( @-@ mm ) .\\n',\n",
              " 'METHODS\\tSecondary outcome measures included the Western Ontario and McMaster Universities Osteoarthritis Index scores , patient global assessment ( PGA ) of the severity of knee OA , and @-min walk distance ( @MWD ) .\\n',\n",
              " 'METHODS\\tSerum levels of interleukin @ ( IL-@ ) , IL-@ , tumor necrosis factor ( TNF ) - , and high-sensitivity C-reactive protein ( hsCRP ) were measured .\\n',\n",
              " 'RESULTS\\tThere was a clinically relevant reduction in the intervention group compared to the placebo group for knee pain , physical function , PGA , and @MWD at @ weeks .\\n',\n",
              " 'RESULTS\\tThe mean difference between treatment arms ( @ % CI ) was @ ( @-@ @ ) , p < @ ; @ ( @-@ @ ) , p < @ ; @ ( @-@ @ ) , p < @ ; and @ ( @-@ @ ) , p < @ , respectively .\\n',\n",
              " 'RESULTS\\tFurther , there was a clinically relevant reduction in the serum levels of IL-@ , IL-@ , TNF - , and hsCRP at @ weeks in the intervention group when compared to the placebo group .\\n',\n",
              " 'RESULTS\\tThese differences remained significant at @ weeks .\\n',\n",
              " 'RESULTS\\tThe Outcome Measures in Rheumatology Clinical Trials-Osteoarthritis Research Society International responder rate was @ % in the intervention group and @ % in the placebo group ( p < @ ) .\\n',\n",
              " 'CONCLUSIONS\\tLow-dose oral prednisolone had both a short-term and a longer sustained effect resulting in less knee pain , better physical function , and attenuation of systemic inflammation in older patients with knee OA ( ClinicalTrials.gov identifier NCT@ ) .\\n',\n",
              " '\\n',\n",
              " '###24854809\\n',\n",
              " 'BACKGROUND\\tEmotional eating is associated with overeating and the development of obesity .\\n',\n",
              " 'BACKGROUND\\tYet , empirical evidence for individual ( trait ) differences in emotional eating and cognitive mechanisms that contribute to eating during sad mood remain equivocal .\\n',\n",
              " 'OBJECTIVE\\tThe aim of this study was to test if attention bias for food moderates the effect of self-reported emotional eating during sad mood ( vs neutral mood ) on actual food intake .\\n',\n",
              " 'OBJECTIVE\\tIt was expected that emotional eating is predictive of elevated attention for food and higher food intake after an experimentally induced sad mood and that attentional maintenance on food predicts food intake during a sad versus a neutral mood .\\n',\n",
              " 'METHODS\\tParticipants ( N = @ ) were randomly assigned to one of the two experimental mood induction conditions ( sad/neutral ) .\\n']"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dEMZl5F4kHF8"
      },
      "source": [
        "#create list of dicts to store data in \n",
        "def preprocess_data(f_name):\n",
        "    \"\"\"\n",
        "    Get information from text file specified.\n",
        "    Read data contained in data list into list of dicts, with every data list entry being represented by a dict.\n",
        "\n",
        "    PARAMS:\n",
        "      f_name(str) - path to file\n",
        "\n",
        "    RETURNS:\n",
        "      lstOdct(lst) - list of dicts\n",
        "    \"\"\"\n",
        "    data = import_data_from_txt(f_name)\n",
        "    line_counter = 0\n",
        "    lstOdct = []\n",
        "    for entry in data:\n",
        "      if entry.startswith(\"###\"):\n",
        "        line_counter = 1\n",
        "        new_abstract = []\n",
        "      elif entry == \"\\n\":\n",
        "        line_counter = 0\n",
        "        #get line count for current abstract\n",
        "        total_lines = len(new_abstract)\n",
        "        #append line count for current abstract to each dict in sublist\n",
        "        new_abstract = [dict(item, **{'total_lines':total_lines}) for item in new_abstract]\n",
        "        #add current abstract entries to lstOdct\n",
        "        lstOdct.extend(new_abstract)\n",
        "      else:\n",
        "        label, sentence = entry.strip().split(\"\\t\")\n",
        "        curr_dict = {\n",
        "            \"line_num\" : line_counter,\n",
        "            \"label\" : label.lower(),\n",
        "            \"text\" : sentence.lower()\n",
        "        }\n",
        "        new_abstract.append(curr_dict)\n",
        "        line_counter  += 1\n",
        "    return lstOdct"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PcBqeqKXoW5p",
        "outputId": "294f1400-b28d-4603-aa60-15709cadafc8"
      },
      "source": [
        "%%time\n",
        "train_dct = preprocess_data('/content/pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/train.txt')\n",
        "test_dct = preprocess_data('/content/pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/test.txt')\n",
        "val_dct = preprocess_data('/content/pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/dev.txt')"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 524 ms, sys: 109 ms, total: 634 ms\n",
            "Wall time: 639 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "i8L3ACiSozL3",
        "outputId": "700aa7f9-744f-42f6-f134-43f26bcb091c"
      },
      "source": [
        "train_dct[:5]"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'label': 'objective',\n",
              "  'line_num': 1,\n",
              "  'text': 'to investigate the efficacy of @ weeks of daily low-dose oral prednisolone in improving pain , mobility , and systemic low-grade inflammation in the short term and whether the effect would be sustained at @ weeks in older adults with moderate to severe knee osteoarthritis ( oa ) .',\n",
              "  'total_lines': 12},\n",
              " {'label': 'methods',\n",
              "  'line_num': 2,\n",
              "  'text': 'a total of @ patients with primary knee oa were randomized @:@ ; @ received @ mg/day of prednisolone and @ received placebo for @ weeks .',\n",
              "  'total_lines': 12},\n",
              " {'label': 'methods',\n",
              "  'line_num': 3,\n",
              "  'text': 'outcome measures included pain reduction and improvement in function scores and systemic inflammation markers .',\n",
              "  'total_lines': 12},\n",
              " {'label': 'methods',\n",
              "  'line_num': 4,\n",
              "  'text': 'pain was assessed using the visual analog pain scale ( @-@ mm ) .',\n",
              "  'total_lines': 12},\n",
              " {'label': 'methods',\n",
              "  'line_num': 5,\n",
              "  'text': 'secondary outcome measures included the western ontario and mcmaster universities osteoarthritis index scores , patient global assessment ( pga ) of the severity of knee oa , and @-min walk distance ( @mwd ) .',\n",
              "  'total_lines': 12}]"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 669
        },
        "id": "FTejjedFrumR",
        "outputId": "f2275473-1098-43e8-c890-56a3cc78c503"
      },
      "source": [
        "#convert list of dicts into DataFrame in order to visualize\n",
        "train_df = pd.DataFrame(train_dct)\n",
        "test_df = pd.DataFrame(test_dct)\n",
        "val_df = pd.DataFrame(val_dct)\n",
        "train_df.head(20)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>line_num</th>\n",
              "      <th>label</th>\n",
              "      <th>text</th>\n",
              "      <th>total_lines</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>objective</td>\n",
              "      <td>to investigate the efficacy of @ weeks of dail...</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>methods</td>\n",
              "      <td>a total of @ patients with primary knee oa wer...</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>methods</td>\n",
              "      <td>outcome measures included pain reduction and i...</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>methods</td>\n",
              "      <td>pain was assessed using the visual analog pain...</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>methods</td>\n",
              "      <td>secondary outcome measures included the wester...</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>6</td>\n",
              "      <td>methods</td>\n",
              "      <td>serum levels of interleukin @ ( il-@ ) , il-@ ...</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>7</td>\n",
              "      <td>results</td>\n",
              "      <td>there was a clinically relevant reduction in t...</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>8</td>\n",
              "      <td>results</td>\n",
              "      <td>the mean difference between treatment arms ( @...</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>9</td>\n",
              "      <td>results</td>\n",
              "      <td>further , there was a clinically relevant redu...</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>10</td>\n",
              "      <td>results</td>\n",
              "      <td>these differences remained significant at @ we...</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>11</td>\n",
              "      <td>results</td>\n",
              "      <td>the outcome measures in rheumatology clinical ...</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>12</td>\n",
              "      <td>conclusions</td>\n",
              "      <td>low-dose oral prednisolone had both a short-te...</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>1</td>\n",
              "      <td>background</td>\n",
              "      <td>emotional eating is associated with overeating...</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>2</td>\n",
              "      <td>background</td>\n",
              "      <td>yet , empirical evidence for individual ( trai...</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>3</td>\n",
              "      <td>objective</td>\n",
              "      <td>the aim of this study was to test if attention...</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>4</td>\n",
              "      <td>objective</td>\n",
              "      <td>it was expected that emotional eating is predi...</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>5</td>\n",
              "      <td>methods</td>\n",
              "      <td>participants ( n = @ ) were randomly assigned ...</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>6</td>\n",
              "      <td>methods</td>\n",
              "      <td>attentional biases for high caloric foods were...</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>7</td>\n",
              "      <td>methods</td>\n",
              "      <td>self-reported emotional eating was assessed wi...</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>8</td>\n",
              "      <td>results</td>\n",
              "      <td>hierarchical multivariate regression modeling ...</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    line_num  ... total_lines\n",
              "0          1  ...          12\n",
              "1          2  ...          12\n",
              "2          3  ...          12\n",
              "3          4  ...          12\n",
              "4          5  ...          12\n",
              "5          6  ...          12\n",
              "6          7  ...          12\n",
              "7          8  ...          12\n",
              "8          9  ...          12\n",
              "9         10  ...          12\n",
              "10        11  ...          12\n",
              "11        12  ...          12\n",
              "12         1  ...          11\n",
              "13         2  ...          11\n",
              "14         3  ...          11\n",
              "15         4  ...          11\n",
              "16         5  ...          11\n",
              "17         6  ...          11\n",
              "18         7  ...          11\n",
              "19         8  ...          11\n",
              "\n",
              "[20 rows x 4 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_smxIcMdsYkY"
      },
      "source": [
        "# check what the distribution of abstract length is\n",
        "len_distrib_sorted = train_df.total_lines.value_counts().sort_values(ascending=False).to_frame()"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jb2y53jou7kL"
      },
      "source": [
        "# check what the distribution of abstract length is\n",
        "len_distrib = train_df.total_lines.value_counts().to_frame().sort_index()"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "qAEdGW7ct6tH",
        "outputId": "bd607231-73dc-4bc3-e926-cfefbb48dbe1"
      },
      "source": [
        "train_df.total_lines.plot.hist(bins=len(len_distrib))\n",
        "plt.show()"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAD4CAYAAAAtrdtxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAUDElEQVR4nO3dfdCddX3n8ffHgCuiXUJJWZaEBm2mLrWKeIvsaHepTjHAtMFdl4U+kLqMccYwo9P+QXR2NqzWGdxZcZcdZYtjxtAVIxWV7BKXpgzW9g8eAlIey5BiWBIjSY0aqB1Y8Lt/nN+NZ8N9J+e+cp9zcu68XzNn7ut8z/Xw+8015MN1/a6HVBWSJHXxinE3QJI0uQwRSVJnhogkqTNDRJLUmSEiSersmHE3YNROOumkWr58+bibIUkT5d577/27qlpyYP2oC5Hly5ezbdu2cTdDkiZKkidnqns6S5LU2dBCJMmyJHckeSTJw0k+3OpXJdmV5P72uaBvmY8m2Z7ksSTv6auvbLXtSdb11U9PclerfyXJK4fVH0nSyw3zSOQF4A+r6gzgHGBtkjPab5+pqjPbZwtA++0S4FeAlcDnkixKsgj4LHA+cAZwad96PtXW9UvAD4HLh9gfSdIBhhYiVbW7qu5r088AjwKnHmSRVcCmqnquqr4LbAfObp/tVfVEVT0PbAJWJQnwLuCrbfmNwEXD6Y0kaSYjGRNJshx4C3BXK12R5IEkG5IsbrVTgaf6FtvZarPVfx74UVW9cEB9pu2vSbItyba9e/fOQ48kSTCCEEnyGuBm4CNVtR+4Dng9cCawG/j0sNtQVddX1VRVTS1Z8rIr1CRJHQ31Et8kx9ILkC9V1dcAqurpvt8/D/yv9nUXsKxv8aWtxiz1HwAnJDmmHY30zy9JGoFhXp0V4AvAo1V1TV/9lL7Z3gs81KY3A5ck+UdJTgdWAHcD9wAr2pVYr6Q3+L65es+wvwN4X1t+NXDLsPojSXq5YR6JvAP4PeDBJPe32sfoXV11JlDADuCDAFX1cJKbgEfoXdm1tqpeBEhyBXAbsAjYUFUPt/VdCWxK8kfAd+iFliRpRHK0vZRqamqqvGP9yLZ83a0Dz7vj6guH2BJJ05LcW1VTB9a9Y12S1JkhIknqzBCRJHVmiEiSOjNEJEmdGSKSpM6OupdSaTzmctmupMnhkYgkqTNDRJLUmSEiSerMEJEkdWaISJI6M0QkSZ0ZIpKkzgwRSVJnhogkqTNDRJLUmSEiSerMEJEkdeYDGDXR5vpgR9/JLs0vj0QkSZ0ZIpKkzgwRSVJnhogkqTNDRJLUmSEiSerMEJEkdWaISJI6M0QkSZ0ZIpKkzgwRSVJnhogkqTNDRJLU2dBCJMmyJHckeSTJw0k+3OonJtma5PH2d3GrJ8m1SbYneSDJWX3rWt3mfzzJ6r76W5M82Ja5NkmG1R9J0ssN80jkBeAPq+oM4BxgbZIzgHXA7VW1Ari9fQc4H1jRPmuA66AXOsB64O3A2cD66eBp83ygb7mVQ+yPJOkAQwuRqtpdVfe16WeAR4FTgVXAxjbbRuCiNr0KuKF67gROSHIK8B5ga1Xtq6ofAluBle23n6uqO6uqgBv61iVJGoGRjIkkWQ68BbgLOLmqdrefvg+c3KZPBZ7qW2xnqx2svnOGuiRpRIYeIkleA9wMfKSq9vf/1o4gagRtWJNkW5Jte/fuHfbmJOmoMdQQSXIsvQD5UlV9rZWfbqeiaH/3tPouYFnf4ktb7WD1pTPUX6aqrq+qqaqaWrJkyeF1SpL0kmFenRXgC8CjVXVN30+bgekrrFYDt/TVL2tXaZ0D/Lid9roNOC/J4jagfh5wW/ttf5Jz2rYu61uXJGkEjhniut8B/B7wYJL7W+1jwNXATUkuB54ELm6/bQEuALYDPwHeD1BV+5J8ArinzffxqtrXpj8EfBE4Dvhm+0iSRmRoIVJVfwXMdt/Gu2eYv4C1s6xrA7Bhhvo24I2H0UxJ0mEY5pGIFrjl624ddxMkjZmPPZEkdWaISJI6M0QkSZ0ZIpKkzgwRSVJnhogkqTNDRJLUmfeJ6Kgyl3tbdlx94RBbIi0MHolIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktSZISJJ6mygEEnyq8NuiCRp8gx6JPK5JHcn+VCSfzzUFkmSJsZAIVJVvwb8DrAMuDfJjUl+42DLJNmQZE+Sh/pqVyXZleT+9rmg77ePJtme5LEk7+mrr2y17UnW9dVPT3JXq38lySvn0G9J0jwYeEykqh4H/j1wJfAvgWuT/E2SfzXLIl8EVs5Q/0xVndk+WwCSnAFcAvxKW+ZzSRYlWQR8FjgfOAO4tM0L8Km2rl8CfghcPmhfJEnzY9AxkTcl+QzwKPAu4Der6p+16c/MtExVfRvYN2A7VgGbquq5qvousB04u322V9UTVfU8sAlYlSRt219ty28ELhpwW5KkeTLokch/A+4D3lxVa6vqPoCq+h69o5O5uCLJA+101+JWOxV4qm+ena02W/3ngR9V1QsH1GeUZE2SbUm27d27d47NlSTNZtAQuRC4sar+ASDJK5K8GqCq/mQO27sOeD1wJrAb+PQclu2sqq6vqqmqmlqyZMkoNilJR4VBQ+TPgeP6vr+61eakqp6uqher6qfA5+mdrgLYRW/QftrSVput/gPghCTHHFCXJI3QMYeeBYBXVdWz01+q6tnpI5G5SHJKVe1uX98LTF+5tRm4Mck1wD8FVgB3AwFWJDmdXkhcAvx2VVWSO4D30RsnWQ3cMtf2SAezfN2tA8+74+oLh9gS6cg1aIj8fZKzpsdCkrwV+IeDLZDky8C5wElJdgLrgXOTnAkUsAP4IEBVPZzkJuAR4AVgbVW92NZzBXAbsAjYUFUPt01cCWxK8kfAd4AvDNgXSdI8GTREPgL8aZLv0Ts6+CfAvz3YAlV16QzlWf+hr6pPAp+cob4F2DJD/Ql+djpMkjQGA4VIVd2T5A3AL7fSY1X1f4fXLEnSJBj0SATgbcDytsxZSaiqG4bSKknSRBgoRJL8Cb1Lc+8HXmzlAgwRSTqKDXokMgWcUVU1zMZIkibLoPeJPERvMF2SpJcMeiRyEvBIkruB56aLVfVbQ2mVJGkiDBoiVw2zEZKkyTToJb5/keQXgRVV9eftbvVFw22aJOlIN+ij4D9A77Hrf9xKpwLfGFajJEmTYdCB9bXAO4D98NILqn5hWI2SJE2GQUPkufZSKADa03O93FeSjnKDhshfJPkYcFx7t/qfAv9zeM2SJE2CQUNkHbAXeJDek3e3MPc3GkqSFphBr86afonU54fbHEnSJBn02VnfZYYxkKp63by3SJI0Meby7KxprwL+DXDi/DdHkjRJBhoTqaof9H12VdV/AXwfqCQd5QY9nXVW39dX0Dsymcu7SDQB5vJOcUmCwYPg033TL9B7P/rF894aSdJEGfTqrF8fdkMkSZNn0NNZf3Cw36vqmvlpjiRpkszl6qy3AZvb998E7gYeH0ajJEmTYdAQWQqcVVXPACS5Cri1qn53WA2TJB35Bn3sycnA833fn281SdJRbNAjkRuAu5N8vX2/CNg4nCZJkibFoFdnfTLJN4Ffa6X3V9V3htcsSdIkGPR0FsCrgf1V9V+BnUlOH1KbJEkTYtDX464HrgQ+2krHAv9jWI2SJE2GQY9E3gv8FvD3AFX1PeC1w2qUJGkyDBoiz1dV0R4Hn+T44TVJkjQpBr0666YkfwyckOQDwL/DF1RJL5nLwyt3XO0DsLVwHDJEkgT4CvAGYD/wy8B/qKqtQ26bJOkId8gQqapKsqWqfhUwOCRJLxl0TOS+JG+by4qTbEiyJ8lDfbUTk2xN8nj7u7jVk+TaJNuTPND//pIkq9v8jydZ3Vd/a5IH2zLXtiMmSdIIDRoibwfuTPK37R/5B5M8cIhlvgisPKC2Dri9qlYAt7fvAOcDK9pnDXAd9EIHWN+2fzawfjp42jwf6FvuwG1JkobsoKezkpxWVf8HeM9cV1xV306y/IDyKuDcNr0R+Ba9+09WATe0K8DuTHJCklPavFural9rz1ZgZZJvAT9XVXe2+g30HsXyzbm2U5LU3aHGRL5B7+m9Tya5uar+9WFu7+Sq2t2mv8/PHuJ4KvBU33w7W+1g9Z0z1GeUZA29IxxOO+20w2i+JKnfoU5n9Y8zvG4+N9x/38mwVdX1VTVVVVNLliwZxSYl6ahwqBCpWaa7erqdpqL93dPqu4BlffMtbbWD1ZfOUJckjdChQuTNSfYneQZ4U5ven+SZJPs7bG8zMH2F1Wrglr76Ze0qrXOAH7fTXrcB5yVZ3AbUzwNua7/tT3JOuyrrsr51SZJG5KBjIlW1qOuKk3yZ3sD4SUl20rvK6mp6d79fDjwJXNxm3wJcAGwHfgK8v21/X5JPAPe0+T4+PcgOfIjeFWDH0RtQd1BdkkZs0MeezFlVXTrLT++eYd4C1s6yng3Ahhnq24A3Hk4bJUmHZy7vE5Ek6f9jiEiSOjNEJEmdGSKSpM4MEUlSZ4aIJKkzQ0SS1JkhIknqzBCRJHVmiEiSOjNEJEmdGSKSpM4MEUlSZ4aIJKkzQ0SS1JkhIknqbGgvpdKRYfm6W8fdBB1grvtkx9UXDqkl0uHzSESS1JkhIknqzBCRJHVmiEiSOjNEJEmdGSKSpM4MEUlSZ4aIJKkzQ0SS1JkhIknqzBCRJHVmiEiSOjNEJEmdGSKSpM4MEUlSZ4aIJKmzsYRIkh1JHkxyf5JtrXZikq1JHm9/F7d6klybZHuSB5Kc1bee1W3+x5OsHkdfJOloNs4jkV+vqjOraqp9XwfcXlUrgNvbd4DzgRXtswa4DnqhA6wH3g6cDayfDh5J0mgcSa/HXQWc26Y3At8Crmz1G6qqgDuTnJDklDbv1qraB5BkK7AS+PJomy0N11xep+urdDVq4zoSKeDPktybZE2rnVxVu9v094GT2/SpwFN9y+5stdnqL5NkTZJtSbbt3bt3vvogSUe9cR2JvLOqdiX5BWBrkr/p/7GqKknN18aq6nrgeoCpqal5W68kHe3GciRSVbva3z3A1+mNaTzdTlPR/u5ps+8ClvUtvrTVZqtLkkZk5CGS5Pgkr52eBs4DHgI2A9NXWK0GbmnTm4HL2lVa5wA/bqe9bgPOS7K4Daif12qSpBEZx+msk4GvJ5ne/o1V9b+T3APclORy4Eng4jb/FuACYDvwE+D9AFW1L8kngHvafB+fHmSXJI3GyEOkqp4A3jxD/QfAu2eoF7B2lnVtADbMdxslSYPxjnVJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTOjqRHwWtAc3k0uI4uPjZeo+aRiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTOvE9EOkrN9X4j7yvRTDwSkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktSZISJJ6swQkSR15h3rkgbiWxM1E0PkCODrbiVNKk9nSZI6M0QkSZ15OkvSvBvWKVrHWo48HolIkjozRCRJnRkikqTOJj5EkqxM8liS7UnWjbs9knQ0meiB9SSLgM8CvwHsBO5JsrmqHhlvy7z3QxqGI+W/Kwf4f2aiQwQ4G9heVU8AJNkErALGHiKSFq5h3r0/aU8GmPQQORV4qu/7TuDtB86UZA2wpn19NsljI2jbScDfjWA747TQ+2j/Jt/Y+5hPDXXdo+zfL85UnPQQGUhVXQ9cP8ptJtlWVVOj3OaoLfQ+2r/Jt9D7eCT0b9IH1ncBy/q+L201SdIITHqI3AOsSHJ6klcClwCbx9wmSTpqTPTprKp6IckVwG3AImBDVT085mZNG+npszFZ6H20f5Nvofdx7P1LVY27DZKkCTXpp7MkSWNkiEiSOjNEhiDJjiQPJrk/ybZxt+dwJdmQZE+Sh/pqJybZmuTx9nfxONt4uGbp41VJdrX9eH+SC8bZxsORZFmSO5I8kuThJB9u9QWxHw/SvwWxD5O8KsndSf669e8/tvrpSe5qj336SrvAaLRtc0xk/iXZAUxV1YK4kSvJvwCeBW6oqje22n8C9lXV1e2ZZYur6spxtvNwzNLHq4Bnq+o/j7Nt8yHJKcApVXVfktcC9wIXAb/PAtiPB+nfxSyAfZgkwPFV9WySY4G/Aj4M/AHwtaralOS/A39dVdeNsm0eieiQqurbwL4DyquAjW16I73/YCfWLH1cMKpqd1Xd16afAR6l98SHBbEfD9K/BaF6nm1fj22fAt4FfLXVx7L/DJHhKODPktzbHrmyEJ1cVbvb9PeBk8fZmCG6IskD7XTXRJ7qOVCS5cBbgLtYgPvxgP7BAtmHSRYluR/YA2wF/hb4UVW90GbZyRiC0xAZjndW1VnA+cDadqpkwareOdGFeF70OuD1wJnAbuDT423O4UvyGuBm4CNVtb//t4WwH2fo34LZh1X1YlWdSe/JHGcDbxhzkwBDZCiqalf7uwf4Or0dvtA83c5DT5+P3jPm9sy7qnq6/Yf7U+DzTPh+bOfSbwa+VFVfa+UFsx9n6t9C24cAVfUj4A7gnwMnJJm+aXwsj30yROZZkuPbwB5JjgfOAx46+FITaTOwuk2vBm4ZY1uGYvof1+a9TPB+bAOzXwAerapr+n5aEPtxtv4tlH2YZEmSE9r0cfTeofQovTB5X5ttLPvPq7PmWZLX0Tv6gN5jZW6sqk+OsUmHLcmXgXPpPVb7aWA98A3gJuA04Eng4qqa2IHpWfp4Lr3TIAXsAD7YN34wUZK8E/hL4EHgp638MXrjBhO/Hw/Sv0tZAPswyZvoDZwvovc//zdV1cfbvzebgBOB7wC/W1XPjbRthogkqStPZ0mSOjNEJEmdGSKSpM4MEUlSZ4aIJKkzQ0SS1JkhIknq7P8BoGQNQdKx+KoAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "2Y5rlLtQsz4q",
        "outputId": "917c15f2-efa1-44bd-b7ab-aaf90c883716"
      },
      "source": [
        "len_distrib.plot()\n",
        "plt.show()"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD4CAYAAAAO9oqkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU9b3/8ddnJvsGSQiBJKwS1rAEIoIWd2URZXFp7a1if23tveqt3vb2V2xt3Wrrtdu1v1Z7bfVWu7lUFIogUIoLVZSAARK2BETIHhLISraZ7++POdERE7JNcmb5PB+PeWTmO2dmPofRvHO+3+/5HjHGoJRSKrQ57C5AKaWU/TQMlFJKaRgopZTSMFBKKYWGgVJKKSDM7gL6atiwYWbs2LF2l6GUUgFl165dJ40xKWe3B2wYjB07ltzcXLvLUEqpgCIiH3XWrt1ESimlug8DERklIttEZL+IFIjI3Vb7AyJSIiJ51m2J12vuFZEiETkkIgu92hdZbUUistqrfZyIvGe1vyAiEb7eUaWUUl3ryZFBO/AtY8xUYB5wp4hMtZ77hTFmlnXbAGA99wVgGrAIeEJEnCLiBH4NLAamAjd7vc9/We81ATgFfMVH+6eUUqoHuh0zMMaUAWXW/XoROQCkn+Mly4DnjTEtwIciUgTMtZ4rMsYcBRCR54Fl1vtdDnzR2uZZ4AHgyd7vjlIqkLS1tVFcXExzc7PdpQSdqKgoMjIyCA8P79H2vRpAFpGxQDbwHnARcJeI3Ark4jl6OIUnKHZ4vayYT8LjxFntFwDJwGljTHsn25/9+bcDtwOMHj26N6UrpfxQcXEx8fHxjB07FhGxu5ygYYyhurqa4uJixo0b16PX9HgAWUTigJeBe4wxdXj+cj8PmIXnyOFnvS+5d4wxTxljcowxOSkpn5kZpZQKMM3NzSQnJ2sQ+JiIkJyc3Ksjrh4dGYhIOJ4g+JMxZg2AMabC6/nfAuuthyXAKK+XZ1htdNFeDQwVkTDr6MB7e6VUkNMgGBi9/XftyWwiAZ4GDhhjfu7VPtJrsxVAvnV/HfAFEYkUkXFAJvA+sBPItGYOReAZZF5nPGtobwNusF6/Cljbq71QfuedopO8cajS7jKUUj3UkyODi4BbgH0ikme1fRfPbKBZgAGOAV8HMMYUiMiLwH48M5HuNMa4AETkLmAT4ASeMcYUWO/3HeB5Efkh8AGe8FEB6ERNEw+v38/m/RWEO4VN91zM+JQ4u8tSSnWj2yMDY8x2Y4wYY2Z4TyM1xtxijJlutV9nzTrqeM0jxpjzjDGTjDEbvdo3GGMmWs894tV+1Bgz1xgzwRhzozUTSQWQ5jYXj/+9kCt//iZvF57kG1dkEhXm5MG/7UcvoKT81enTp3niiSfOuc2xY8f485//3O17HTt2jKysrC6ff+ONN1i6dCkA69at49FHH+1dsQNMz0BW/WKMYcv+Cq76xZv84u+HuXJqKlu/dQnfvGoid1+ZyZuHq/j7Ae0uUv7Jl2HQG9dddx2rV6/ufsNBFLBrEyn7HTvZyIN/K2DboSoyh8fx569ewIUThn38/KoLx/LCzhM8tL6ABZnDiAp32lit8ncP/q2A/aV1Pn3PqWkJ3H/ttC6fX716NUeOHGHWrFlcddVVAGzcuBER4b777uPzn/88q1ev5sCBA8yaNYtVq1axYsUKbrnlFhobGwH41a9+xYUXXtirun7/+9+Tm5vLr371K2677TYSEhLIzc2lvLycxx57jBtu8Ayh/uQnP+HFF1+kpaWFFStW8OCDD9LY2MhNN91EcXExLpeL73//+3z+85/v47/QJzQMVK81tbbzxLYjPPXWUSLCHNx3zRRWXTiWcOenDzTDnQ4evG4aX/zdezz11lG+cUWmTRUr1blHH32U/Px88vLyePnll/nNb37Dnj17OHnyJOeffz4XX3wxjz76KD/96U9Zv94zYbKpqYktW7YQFRVFYWEhN998c78XzSwrK2P79u0cPHiQ6667jhtuuIHNmzdTWFjI+++/jzGG6667jrfeeouqqirS0tJ47bXXAKitre33vwNoGKheMMbwen45D6/fT2ltMyuz01m9eDLDE6K6fM2FE4ZxzfSR/HpbEStnp5ORGDOIFatAcq6/4AfD9u3bufnmm3E6naSmpnLJJZewc+dOEhISPrVdW1sbd911F3l5eTidTg4fPtzvz16+fDkOh4OpU6dSUeGZtb9582Y2b95MdnY2AA0NDRQWFrJgwQK+9a1v8Z3vfIelS5eyYMGCfn8+aBioXvj3v3zA+r1lTB4Rz+M3Z3P+2KQeve6710xh68EKHnntAE9+ac4AV6nUwPrFL35Bamoqe/bswe12ExXV9R9DPRUZGfnx/Y4JF8YY7r33Xr7+9a9/Zvvdu3ezYcMG7rvvPq644gp+8IMf9LsGHUBWPXK0qoH1e8v4PxeNY/2/f67HQQCQPjSauy6bwMb8crYXnhzAKpXqnfj4eOrr6wFYsGABL7zwAi6Xi6qqKt566y3mzp37qW3A0y0zcuRIHA4Hf/jDH3C5XANS28KFC3nmmWdoaGgAoKSkhMrKSkpLS4mJieFLX/oS3/72t9m9e7dPPk+PDFSPbMwvB+CrC8YR5uz93xBfXTCeF3OLuX9dPhvvvpiIMP07RNkvOTmZiy66iKysLBYvXsyMGTOYOXMmIsJjjz3GiBEjSE5Oxul0MnPmTG677TbuuOMOrr/+ep577jkWLVpEbGzsgNR29dVXc+DAAebPnw9AXFwcf/zjHykqKuLb3/42DoeD8PBwnnzSN2t6SqDOAc/JyTF6pbPBc80v3yYizMErd1zU5/fYeqCCrzyby/eWTOFrF4/3YXUqUB04cIApU6bYXUbQ6uzfV0R2GWNyzt5W/zxT3fqoupGC0jqWZI3sfuNzuGJKKpdPHs7jWwuprNMli5XyJxoGqlsdXUSLskb0+71+sHQqre1uHt14sN/vpZQ/2rRpE7NmzfrUbcWKFXaX1S0dM1Dd2rivjJkZQxiV1P9poWOHxfK1i8fx621H+OIFo8npxUC0Ck7GmKBauXThwoUsXLiw+w0HWG+HAPTIQJ3TiZom9hTXsnh6/7qIvN152QRGDoniB2sLcLkDc8xK+UZUVBTV1dW6fpWPdVzcpjfTXvXIQJ3T61YX0WIfdBF1iIkI43vXTOGuP3/An98/zi3zxvjsvVVgycjIoLi4mKqqKrtLCTodl73sKQ0DdU4b8suYlpbAmGTfTp+7ZvpI/jT+OD/bfIil00eSGBvh0/dXgSE8PLzHl2VUA0u7iVSXSk+f4YPjp1niwy6iDiLCA9dNo765nZ9sPuTz91dK9Y6GgerSQHQReZs0Ip5b54/hL+8fJ7/EN4ttKaX6RsNAdWljvmcdooG8Utk9V04kOTaCH6zNx62DyUrZRsNAdaqirpncj04NSBeRtyHR4fzn1ZPYffw0Oz6sHtDPUkp1TcNAder1/HKMgSXTB6aLyNvy7HTio8L4a27xgH+WUqpzGgaqUxv2lZE5PI4Jw+MH/LOiwp1cOzONDfll1De3DfjnKaU+S8NAfUZVfQvvH6vx6Ylm3blxTgbNbW5e21s2aJ+plPqEhoH6jE0Fg9dF1GHWqKFMGB7Hi7knBu0zlVKf0DBQn7Exv4zxKbFMSh34LqIOIsKNczLYffw0RZUNg/a5SikPDQP1KdUNLew4WsOSrJGDvnjYitnpOB3CX3fpQLJSg03DQH3Klv0VuNyGxYPYRdRheHwUl05MYc3uYtpd7kH/fKVCmYaB+pTX9pUxJjmGqSMTbPn8G3MyqKxv4W29VrJSg0rDQH3sVGMr7xypZrENXUQdLp+cSlJsBC/t0oFkpQaThoH62JYDni6iawZxSunZIsIcLJuVxt/3V3KqsdW2OpQKNRoG6mMb95WRkRhNVro9XUQdbpwzilaXm7V5JbbWoVQo0TBQANSeaWN70UmWTLevi6jD1LQEpqUl8JLOKlJq0GgYKAC2HqigzWUGbLnq3ropZxQFpXXsL62zuxSlQkK3YSAio0Rkm4jsF5ECEbnbak8SkS0iUmj9TLTaRUR+KSJFIrJXRGZ7vdcqa/tCEVnl1T5HRPZZr/ml2P2naQjasK+ctCFRzBo11O5SAFg2K40Ip0MHkpUaJD05MmgHvmWMmQrMA+4UkanAamCrMSYT2Go9BlgMZFq324EnwRMewP3ABcBc4P6OALG2+ZrX6xb1f9dUT9U3t/FWYRWLbJxFdLahMRFcNTWVtXmltLbrOQdKDbRuw8AYU2aM2W3drwcOAOnAMuBZa7NngeXW/WXAc8ZjBzBUREYCC4EtxpgaY8wpYAuwyHouwRizwxhjgOe83ksNgn8crKS13c01M/yji6jDDTkZ1DS28o+DFXaXolTQ69WYgYiMBbKB94BUY0zHEpPlQKp1Px3wPrYvttrO1V7cSXtnn3+7iOSKSG5VVVVvSlfnsGFfGakJkWSPSux+40F0cWYKqQmRvKTXOVBqwPU4DEQkDngZuMcY86lRPesv+gG/ZqEx5iljTI4xJiclJWWgPy4kNLa088ahKhZnjcTh8I8uog5Oh7BydgbbDlVSWddsdzlKBbUehYGIhOMJgj8ZY9ZYzRVWFw/Wz0qrvQQY5fXyDKvtXO0ZnbSrQbDtUCUt7W6/mUV0thvnZOA2sOYD/U9CqYHUk9lEAjwNHDDG/NzrqXVAx4ygVcBar/ZbrVlF84BaqztpE3C1iCRaA8dXA5us5+pEZJ71Wbd6vZcaYBv3lTMsLpKcsUl2l9Kp8SlxzBmTyEu5J/AcgCqlBkJPjgwuAm4BLheRPOu2BHgUuEpECoErrccAG4CjQBHwW+AOAGNMDfAwsNO6PWS1YW3zO+s1R4CNPtg31Y0zrS7+cbCSRVmpOP2si8jbjXMyOFLVyAcnTttdilJBK6y7DYwx24GuflNc0cn2Brizi/d6Bnimk/ZcIKu7WpRv7T5+ijNtLq6cktr9xja6ZsZIHvhbAS/lFjN7tH8NcisVLPQM5BCWX1ILwMwM/zjRrCvxUeEsyRrJ+j2lnGl12V2OUkFJwyCE5ZfWkT40msTYCLtL6dYNORnUt7SzqaDc7lKUCkoaBiEsv6SW6elD7C6jR+aNSyYjMVqXp1BqgGgYhKj65jY+PNlo+3LVPeVwCDfMyeCdI9UUn2qyuxylgo6GQYjqWA10WoAcGQBcPzsDY+DlXXrOgVK+pmEQovKtMMhKC5wwGJUUw0UTkvnr7hO43XrOgVK+pGEQogpKaklNiCQlPtLuUnrlxjmjOFFzhvc+rOl+Y6VUj2kYhKj80tqAOirosHDaCOIiw1izWxevU8qXNAxCUFNrO0WVDQE1XtAhOsLJ4qwRbNhXpuccKOVDGgYh6EBZPW5DwEwrPduK2ek0trrYvF/POVDKVzQMQlBBqefM40CZVnq2eeOSSRsSxSu6kqlSPqNhEILyS2pJjo1gREKU3aX0icMhLM9O5+3Ck1TW63UOlPIFDYMQlF9Sx7T0IX5zveO+WDk7HZfbsC6v1O5SlAoKGgYhpqXdxeGKerLSArOLqMOE4fFMTx+iXUVK+YiGQYg5VF5Pu9uQFaCDx95Wzk6noLSOwxX1dpeiVMDTMAgx+SWBd+ZxV66dmYbTIazZrUcHSvWXhkGIyS+tJSEqjFFJ0XaX0m/D4iK5ZGIKr35QgkuXp1CqXzQMQkxBSS1ZAT547G1Fdjrldc3sOFptdylKBTQNgxDS5nJzoLw+KMYLOlw1NZX4yDDtKlKqnzQMQkhRZQOt7W6mBfhMIm9R4U6WTB/J6/m6PIVS/aFhEEL2lXSceRw8Rwagy1Mo5QsaBiGkoKSW2Agn45Jj7S7Fp+aOTSJ9aDQva1eRUn2mYRBC8kvrmJY2BIcjOAaPO3iWp0hje2EVlXW6PIVSfaFhECJcbsP+0jqmBejidN1ZkZ2B28C6Pbo8hVJ9oWEQIj482cCZNldQnGzWmQnD45iZMURnFSnVRxoGIeLjM4+DbPDY24rsdPaX1XGwvM7uUpQKOBoGISK/pJbIMAfnpQTX4LG3a2emEeYQXtGjA6V6TcMgROwrqWXKyATCnMH7lSfHRXLppBRezdPlKZTqreD9zaA+5rYGjwP1yma9sSI7g4q6Ft49ostTKNUbGgYh4HhNE/Ut7QF7zePeuGLKcOKjwlizu9juUpQKKBoGISDfuubxtCCdSeQtKtzJNdNH8npBOU2t7XaXo1TA0DAIAfkldYQ7hYmp8XaXMihWzs6gqdXFpgJdnkKpnuo2DETkGRGpFJF8r7YHRKRERPKs2xKv5+4VkSIROSQiC73aF1ltRSKy2qt9nIi8Z7W/ICIRvtxBBQWltUwaEU9EWGhkf86YRDISo/WcA6V6oSe/HX4PLOqk/RfGmFnWbQOAiEwFvgBMs17zhIg4RcQJ/BpYDEwFbra2Bfgv670mAKeAr/Rnh9SnGWPYV1IbtCebdcbhEFZkp/PPopNU6PIUSvVIt2FgjHkLqOnh+y0DnjfGtBhjPgSKgLnWrcgYc9QY0wo8DywTzxVWLgf+ar3+WWB5L/dBnUPJ6TOcbmpjWggMHntbkZ2O28DaPD06UKon+tNvcJeI7LW6kRKttnTghNc2xVZbV+3JwGljTPtZ7Z0SkdtFJFdEcquqqvpReuj45JrHwT+t1Nv4lDhmjRqqXUVK9VBfw+BJ4DxgFlAG/MxnFZ2DMeYpY0yOMSYnJSVlMD4y4BWU1uJ0CFNGhlYYAKycnc7B8noOlOnyFEp1p09hYIypMMa4jDFu4Ld4uoEASoBRXptmWG1dtVcDQ0Uk7Kx25SP5JbVkDo8jKtxpdymDbukMz/IUes6BUt3rUxiIyEivhyuAjplG64AviEikiIwDMoH3gZ1ApjVzKALPIPM6Y4wBtgE3WK9fBaztS02qcx3XMAhFSbERXDppOGvzSnV5CqW60ZOppX8B3gUmiUixiHwFeExE9onIXuAy4D8AjDEFwIvAfuB14E7rCKIduAvYBBwAXrS2BfgO8E0RKcIzhvC0T/cwhFXUNVNV3xISy1B0ZeXsdCrrW3jnyEm7S1HKr4V1t4Ex5uZOmrv8hW2MeQR4pJP2DcCGTtqP8kk3k/Kh/CC95nFvXD7ZszzFK7tLWJCp40xKdSU0zkIKUfkldYgQkoPHHaLCnSydoctTKNUdDYMgll9ay7hhscRFdnsAGNSWz0rX5SmU6oaGQRArKKkNiZVKu3P+2CTSh+ryFEqdi4ZBkKpuaKG0tjmklqHoivfyFJW6PIVSndIwCFL5pZ4TraaF8Ewibytme5anWLen1O5SlPJLGgZBqmMmUaieY3C281LimJkxRLuKlOqChkGQKiitZXRSDEOiw+0uxW+syE5nf1kdh8rr7S5FKb+jYRCk8ktC45rHvXHtzDScDmHNB7o8hVJn0zAIQrVNbRyvadIuorMkx0VyycQU1n6gy1ModTYNgyBUUOYZL9BppZ+1Ijud8rpmdhyttrsUpfyKhkEQKrCuYTAtxK5h0BNXTU0lPjJMB5KVOouGQRDaV1JL2pAokuMi7S7F70SFO1k8fQSv55dxptVldzlK+Q0NgyCUX1obcpe57I0V2Rk0trrYvF+Xp1Cqg4ZBkGloaefDk4165vE5XDDOszzFKx9oV5FSHTQMgsyBsjqMQaeVnoPDISyblcbbhSepqm+xuxyl/IKGQZDZW6zXMOiJFdnpuNxGl6dQyqJhEGTeLqxi3LBYUhOi7C7Fr2WmxpOVnsAregKaUoCGQVA50+ri3SPVXDpJr+jVEyuyM8gvqaOwQpenUErDIIjsOFpNS7ubyyYNt7uUgHCdtTyFDiQrpWEQVLYdqiQ63MnccUl2lxIQUuIjWZA5jLV5pbh1eQoV4jQMgoQxhn8crOSiCclEhTvtLidgrMhOp+T0Gd77sMbuUpSylYZBkDhS1UjxqTNcql1EvXL11BHERjh1IFmFPA2DIPHGoUoAHTzupegIJ4uyRrJxXznNbbo8hQpdGgZBYtuhSiamxpGRGGN3KQFn5ex06lva2bK/wu5SlLKNhkEQaGhp5/0Pa3QWUR/NG5/MiIQonVWkQpqGQRD4Z9FJ2lxGxwv6yOkQlmWn8ebhKqobdHkKFZo0DILAG4cqiYsMI2dsot2lBKyV2Rm43Ia/6fIUKkRpGAQ4YwzbDlaxIHMY4U79Ovtq0oh4po5MYI12FakQpb89AtzB8nrK65p1vMAHbpiTwd7iWvJLau0uRalBp2EQ4LZZU0ov0Sml/Xb9nAyiw5384d2P7C5FqUGnYRDg3jhYxbS0BF2l1AeGRIezPDuNtXtKqG1qs7scpQZVt2EgIs+ISKWI5Hu1JYnIFhEptH4mWu0iIr8UkSIR2Ssis71es8ravlBEVnm1zxGRfdZrfiki4uudDFa1TW3sOn5Ku4h86EvzxtDc5ualXSfsLkWpQdWTI4PfA4vOalsNbDXGZAJbrccAi4FM63Y78CR4wgO4H7gAmAvc3xEg1jZf83rd2Z+luvB2URUut+GyydpF5CvT0oYwZ0wif9jxkS5ep0JKt2FgjHkLOHsVr2XAs9b9Z4HlXu3PGY8dwFARGQksBLYYY2qMMaeALcAi67kEY8wOY4wBnvN6L9WNbQerGBoTzqxROqXUl26dP4aPqpt4u+ik3aUoNWj6OmaQaowps+6XA6nW/XTA+/i62Go7V3txJ+2dEpHbRSRXRHKrqqr6WHpwcLsNbx6u5OLMFJwO7VnzpUVZIxgWF8Ef3j1mdylKDZp+DyBbf9EPyvG0MeYpY0yOMSYnJSW0u0byS2s52dCqXUQDIDLMyRfOH83Wg5WcqGmyuxylBkVfw6DC6uLB+llptZcAo7y2y7DaztWe0Um76sa2g1WIwMWZGgYD4YsXjEaAP7133O5SlBoUfQ2DdUDHjKBVwFqv9lutWUXzgFqrO2kTcLWIJFoDx1cDm6zn6kRknjWL6Fav91LnsO1QJTMzhpIcF2l3KUEpbWg0V01N5YWdx3VpaxUSejK19C/Au8AkESkWka8AjwJXiUghcKX1GGADcBQoAn4L3AFgjKkBHgZ2WreHrDasbX5nveYIsNE3uxa8qhta2FN8WqeUDrBb54/lVFMbr+0t635jpQJcWHcbGGNu7uKpKzrZ1gB3dvE+zwDPdNKeC2R1V4f6xFuFVRiDjhcMsAvPS2Z8SizP7fiI6+dkdP8CpQKYnoEcgLYdrGJYXARZaUPsLiWoiQi3zBvDnhOn2Vt82u5ylBpQGgYBxuU2vHm4iksmDsehU0oH3PVzMoiJ0PWKVPDTMAgweSdOUXumTbuIBklCVDjLs9NZt6eUU42tdpej1IDRMAgw2w5W4XQICyZoGAyWW+ePoaVd1ytSwU3DIMBsO1TJnNGJDIkJt7uUkDF5RAJzxybxxx3Hdb0iFbQ0DAJIRV0zBaV1XKpdRIPulvljOF7TxJuHQ3sZFBW8NAwCyJuHPL+I9PyCwbdw2ghS4iN57t1jdpei1IDQMAgg2w5VMiIhiskj4u0uJeREhDm4ee5o3jhcxfFqXa9IBR8NgwDR5nLzduFJLpucgl7/xx5fnDsahwh/ek+nmargo2EQIHKPnaKhpZ1LtYvINiOGRHH11FReyD2h6xWpoKNhECDeOFRJuFO4aMIwu0sJabfMH8Pppjb+tqfU7lKU8ikNgwCx7VAlc8clERfZ7XJSagDNH59M5vA4/rBDu4pUcNEwCADFp5o4XNGgs4j8gIhwy/wx7C2uJe+ErlekgoeGQQB4w5pSquMF/mFFdjqxEU6dZqqCioZBAPjHwUoyEqM5LyXW7lIUEB8VzsrZGazfW8bJhha7y1HKJzQM/FztmTbeLqzi6qkjdEqpH/nyRWNxuw0/3nDQ7lKU8gkNAz+3ZX8FbS7D0pkj7S5FeRmfEsfXLxnPy7uLeefISbvLUarfNAz83Pq9paQPjSZ71FC7S1Fn+ffLMxmTHMN9r+TreQcq4GkY+LFTja1sLzzJ0hkjtYvID0WFO/nh8iyOnmzkiTeO2F2OUv2iYeDHNhWU0+42LJ2RZncpqgsLMlNYPiuNJ98ooqiywe5ylOozDQM/tn5vGWOSY8hKT7C7FHUO9y2dSkxEGN99ZR/G6PUOVGDSMPBT1Q0tvHPkJNdM1y4ifzcsLpJ7F0/m/Q9reGlXsd3lKNUnGgZ+amN+OW6DdhEFiJtyRnH+2ER+tOEA1XrugQpAGgZ+av3eUsanxDJlpF67IBA4HMKPVkynsaWdR147YHc5SvWahoEfqqxr5r0Pa1g6I027iAJIZmo8/3rJeaz5oIR/Fum5ByqwaBj4oY355RgDS2foiWaB5s7LJjA2OYbvvbJPzz1QAUXDwA+t31vKxNQ4JqZqF1Gg8Zx7MJ1j1U08sa3I7nKU6jENAz9TVnuGncdO6cBxAPtc5jBWZKfz5JtHKKqst7scpXpEw8DPvLa3DNAuokB33zVTiI0M47tr8nG79dwD5f80DPzM+r1lTB2ZwPiUOLtLUf2QHBfJdxdP4f1jNby064Td5SjVLQ0DP3Kipom8E6d1hdIgcWNOBnPHJfGjDQf1ugfK72kY+JEN+6wuouk6XhAMRIQfrciiqVXPPVD+r19hICLHRGSfiOSJSK7VliQiW0Sk0PqZaLWLiPxSRIpEZK+IzPZ6n1XW9oUisqp/uxS41u8tY0bGEEYnx9hdivKRCcPj+bdLzuOVD0p483CV3eUo1SVfHBlcZoyZZYzJsR6vBrYaYzKBrdZjgMVApnW7HXgSPOEB3A9cAMwF7u8IkFBy7GQj+0pqdeA4CN1x2QQmDI/jWy/uobK+2e5ylOrUQHQTLQOete4/Cyz3an/OeOwAhorISGAhsMUYU2OMOQVsARYNQF1+7TWri+ganVIadKLCnfz6i7NpaGnjnufzcOnsIuWH+hsGBtgsIrtE5HarLdUYU2bdLwdSrfvpgPe0imKrrav2zxCR20UkV0Ryq6qC65B7/d4yZo8eSvrQaLtLUQNg0oh4HlqWxTtHqvl//yi0uxylPqO/YfA5Y8xsPF1Ad4rIxd5PGs/i7j77M8gY85QxJscYk5OSkuKrt7XdkaoGDpTV6YlmQe7GORmsnJ3O41sLeUfXLlJ+phQcvXIAAA1sSURBVF9hYIwpsX5WAq/g6fOvsLp/sH5WWpuXAKO8Xp5htXXVHjLW7ylDBJZM1/GCYCYi/HB5FuelxPGN5/N0/ED5lT6HgYjEikh8x33gaiAfWAd0zAhaBay17q8DbrVmFc0Daq3upE3A1SKSaA0cX221hYz1e0s5f0wSI4ZE2V2KGmAxEWE6fqD8Un+ODFKB7SKyB3gfeM0Y8zrwKHCViBQCV1qPATYAR4Ei4LfAHQDGmBrgYWCndXvIagsJhyvqKaxs0BPNQoiOHyh/FNbXFxpjjgIzO2mvBq7opN0Ad3bxXs8Az/S1lkC2fk8pDoHFWRoGoeTGORnsOFrN41sLmTs2iQsnDLO7JBXi9AxkGxljWL+3jHnjk0mJj7S7HDWIdPxA+RsNAxvtL6vj6MlGnUUUonT8QPkTDQMbvba3DKdDWJQ1wu5SlE10/ED5Cw0Dm3R0EV14XjJJsRF2l6NspOcfKH+gYWCTfSW1HK9p4lrtIgp5On6g/IGGgU3W7y0j3CksnKZdRErHD5T9NAxs0O5y89reMhZkpjAkJtzucpSf8B4/eHj9flraXXaXpEKIhoENnvnnh5ScPsPNc0fbXYryMzfOyWDV/DH8/p1jLHn8bXYcrba7JBUiNAwG2YmaJn6+5TBXTknlyinD7S5H+RkR4cFlWfzvbefT0u7mC0/t4Fsv7qGmsdXu0lSQ0zAYRMYY7ns1H6cIDy2bhojYXZLyU5dNHs6W/7iEOy49j7V5JVz+szd4cecJ3DqWoAaIhsEgWrenlDcPV/GfCyeRptctUN2IjnDyfxdNZsPdC8gcHsf/fXkvX3hqB4cr6u0uTQUhDYNBcrqplYf+tp+ZGUO4df5Yu8tRAWRiajwv3D6fx66fweHKepY8/jaPvX6QM606wKx8R8NgkPx4w0FOn2njxytn4HRo95DqHYdDuOn8UWz95iUsm5XOE28c4er/fpNthyq7f7FSPaBhMAjePVLNC7kn+NqC8UxNS7C7HBXAkuMi+dlNM/nL1+YR4XTw5f/dyV1/3k1DS7vdpakAp2EwwJrbXHzvlX2MSorm7isy7S5HBYn55yWz4e4FfPOqiWzML+em37xLea2euaz6TsNggD2xrYijJxt5ZPl0oiOcdpejgkhkmJNvXJHJ06tyOF7TxPJf/5OC0lq7y1IBSsNgABVW1PPkm0dYkZ3OxRNT7C5HBalLJw3npX+djwjc+Jt3+cfBCrtLUgFIw2CAuN2Ge9fsIzYyjPuumWJ3OSrITRmZwKt3XsS4YbF89dlcnnv3mN0lqQCjYTBA/rLzOLkfneJ7S6aQHKdXMVMDLzUhihe/Pp/LJw/nB2sLeHj9fl3wTvWYhsEAqKxr5tGNB5k/Ppkb5mTYXY4KIbGRYfzPLTncduFYnt7+If/6x100tepMI9U9DYMB8ODf9tPS7uZHK6frkhNq0DkdwgPXTeP+a6ey9UAFn/+fHVTW6UwjdW4aBj629UAFr+0r4xuXT2DcsFi7y1Eh7MsXjeO3t+ZwpKqB5b/+JwfL6+wuSfkxDQMfamxp5/uv5jMxNY7bLz7P7nKU4oopqbz49fm0uw03PPkubx2usrsk5ac0DHzoZ5sPU1bXzI9XziAiTP9plX/ISh/Cq3deREZiNF/+/U7+7Y+72LK/gtZ2t92lKT8SZncBwcAYwws7T/D7dz7kXy4YzZwxiXaXpNSnpA2N5q//diH/veUwr+aVsDG/nKTYCK6bmcbK2elMTx+i41shTowJzKlnOTk5Jjc31+4yKD19htVr9vHW4Srmj0/mf26dQ0KUXspS+a82l5u3C6t4eXfJx0cIE4bHsXJ2Ostnpevy6kFORHYZY3I+065h0DfGGF7MPcEP1x/AZQz3Lp7Mv1wwBoeuSKoCSO2ZNjbuK2PN7hLeP1aDCMwfn8zK2RksyhpBXKR2HgQbDQMfKqs9w+qX9/Hm4SrmjU/isetnMjo5xpZalPKV49VNvPJBCWs+KOaj6iaiw52MSorGYXUfOURwODw/RQSHWG3iuVxn2pAols5I4+KJKTpm5sc0DHzAGMNLucU8vH4/7W7D6sWTuWWeHg2o4GKMYffxU6zLK6WqoQW3G9zG4Dae5zruu43BGHC5DS5jKKyo51RTG4kx4SyZPpLl2enMGZ2o/3/4ma7CQI8Be6is9gz3rtnHG4eqmDsuiZ/cMIMxyXoegQo+IsKcMUnMGZPUq9e1tnvGIl7NK+Xl3cX86b3jpA+NZtmsNJZnpzMxNX6AKla+oEcG3TDG8NIu62jAZfjOokncOn+s/rWj1Dk0tLSzuaCcV/NK2V5Yhdt4FtNbPiuNa2em6SC1jfy+m0hEFgGPA07gd8aYR8+1/UCGQZvLTdnpZo7XNPH09qNsO1TF3LFJ/ORGPRpQqreq6ltYv7eUV/NK2XPiNCIwI2MoKXERxESEERvpJCYijJgI56cex0Y4iYn0tIc5BKdDcIgQ5hScIjgcQpjV5nR8couJcBId7tSpsl3w6zAQESdwGLgKKAZ2AjcbY/Z39Zr+hIExhprGVo7XNHHi1BlO1DRxvLqJE6eaOF7TRFlt88erPUaFO/jOosms0qMBpfrtw5ONrM0r4Z2iahpa2mlqbaex1cWZVheNre346tdRRJiDxJhwhkZHMDQmnKEx4STGRDDE+jk0OpyhMREkRIcRHe4kOsJJVJj1M9wTJuFO6VWguNyGNpebNpebdpdnRxwOzwB7R5B1DLg7Hb17b1/y9zGDuUCRMeYogIg8DywDugyDvrrl6ffY9dEpmlpdn2ofFhfJqKRo5oxJZHRSDKMSYxiVFMOkEfEkxUb4ugylQtK4YbHcc+VE7rnys88ZY2huc9PU2k6TFQ6NLZ6gaHO7cbsN7W6D2xqwdrnPulltjS0uTje1crqpjVNNrZw+08aHJxvZ3XSa002ttLl6ljhOhxAV5vg4ICLDHNYvfEOry027y02b65MA6Mtq4U6H96wssUKCj+939pzTIWy652Kiwn175UR/CYN04ITX42LggrM3EpHbgdsBRo8e3acPmpY2hAnD4xiVGOP5pZ8Uw6ikaGIi/OWfQqnQJCJER3j+Ok8eoM8wxtDU6uL0mTZONbZS19xGS5ubM20umttcnGnzhE9Lu5szrdZj67mWdjdhDiHc6SDc6fkZ5nAQHiZEeN0PdzgIc3r+6ncbz4Wu3MYTVh2zr9zGWO3gMmc9dhtr1hbWa4z1mk/eyzkAvRQB9RvQGPMU8BR4uon68h6rF0/2aU1KqcAhIsRGhhEbGUa6DmJ/ir+cGVICjPJ6nGG1KaWUGgT+EgY7gUwRGSciEcAXgHU216SUUiHDL7qJjDHtInIXsAnP1NJnjDEFNpellFIhwy/CAMAYswHYYHcdSikVivylm0gppZSNNAyUUkppGCillNIwUEophZ+sTdQXIlIFfDQIHzUMODkIn2OnYN9H3b/AF+z7OJj7N8YYk3J2Y8CGwWARkdzOFnUKJsG+j7p/gS/Y99Ef9k+7iZRSSmkYKKWU0jDoiafsLmAQBPs+6v4FvmDfR9v3T8cMlFJK6ZGBUkopDQOllFJoGJyTiBwTkX0ikicifbvgsh8RkWdEpFJE8r3akkRki4gUWj8T7ayxv7rYxwdEpMT6HvNEZImdNfaHiIwSkW0isl9ECkTkbqs9KL7Hc+xfUHyHIhIlIu+LyB5r/x602seJyHsiUiQiL1hL+Q9ubTpm0DUROQbkGGOC4mQXEbkYaACeM8ZkWW2PATXGmEdFZDWQaIz5jp119kcX+/gA0GCM+amdtfmCiIwERhpjdotIPLALWA7cRhB8j+fYv5sIgu9QRASINcY0iEg4sB24G/gmsMYY87yI/AbYY4x5cjBr0yODEGKMeQuoOat5GfCsdf9ZPP/jBawu9jFoGGPKjDG7rfv1wAE81xAPiu/xHPsXFIxHg/Uw3LoZ4HLgr1a7Ld+fhsG5GWCziOwSkdvtLmaApBpjyqz75UCqncUMoLtEZK/VjRSQXShnE5GxQDbwHkH4PZ61fxAk36GIOEUkD6gEtgBHgNPGmHZrk2JsCEANg3P7nDFmNrAYuNPqgghaxtNnGIz9hk8C5wGzgDLgZ/aW038iEge8DNxjjKnzfi4YvsdO9i9ovkNjjMsYMwvPtd7nApNtLgnQMDgnY0yJ9bMSeAXPFxdsKqx+2o7+2kqb6/E5Y0yF9T+gG/gtAf49Wn3NLwN/MsassZqD5nvsbP+C7TsEMMacBrYB84GhItJx5ckMoGSw69Ew6IKIxFoDWIhILHA1kH/uVwWkdcAq6/4qYK2NtQyIjl+SlhUE8PdoDUA+DRwwxvzc66mg+B672r9g+Q5FJEVEhlr3o4Gr8IyLbANusDaz5fvT2URdEJHxeI4GwHOt6D8bYx6xsaR+E5G/AJfiWS63ArgfeBV4ERiNZ0nwm4wxATsA28U+Xoqne8EAx4Cve/WvBxQR+RzwNrAPcFvN38XTrx7w3+M59u9mguA7FJEZeAaInXj+GH/RGPOQ9fvmeSAJ+AD4kjGmZVBr0zBQSiml3URKKaU0DJRSSmkYKKWUQsNAKaUUGgZKKaXQMFBKKYWGgVJKKeD/A/vjpVjNuPz4AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RSrZ64gxvZL3"
      },
      "source": [
        "# check what the distribution of abstract labels is\n",
        "label_distrib = train_df.label.value_counts()"
      ],
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 314
        },
        "id": "A-0SQz-qvgdl",
        "outputId": "2b15f7ad-4a0c-4c81-92e2-275ede4ea718"
      },
      "source": [
        "label_distrib.plot(kind=\"bar\")\n",
        "plt.show() \n"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEpCAYAAABx406aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAc3ElEQVR4nO3de7hcVZ3m8e9Lwk0UEiSdwYQmaY04kVEuAeJ1emQMAdQwjiC0LWlMk3lGVByd7oZpn6ZF7QHb1hEfxUYJBkdFwLaJXAwZhMEbl0NArtKc4SKJQCIJF6EVwXf+2KtI5XAudUhSq0i9n+epp/Zee1ed3ynIeWvvvfZask1ERPS3bWoXEBER9SUMIiIiYRAREQmDiIggYRAREcDE2gU8X7vttptnzJhRu4yIiBeMG2644Ve2pwy37QUbBjNmzGBgYKB2GRERLxiS7htpW04TRUREwiAiIhIGERFBh2EgaZKkCyX9XNIdkl4naVdJKyTdVZ4nl30l6QxJg5JulrRf2/ssLPvfJWlhW/v+km4przlDkjb/rxoRESPp9Mjg88D3bb8KeC1wB3AScIXtWcAVZR3gUGBWeSwGzgSQtCtwCnAQcCBwSitAyj7Ht71u/qb9WhERMR5jhoGkXYA3A2cD2H7K9iPAAmBp2W0pcERZXgCc68Y1wCRJuwOHACtsr7O9HlgBzC/bdrZ9jZtR885te6+IiOiCTo4MZgJrgXMk3Sjpq5J2AqbafqDs8yAwtSxPA+5ve/2q0jZa+6ph2p9D0mJJA5IG1q5d20HpERHRiU7CYCKwH3Cm7X2BJ9hwSgiA8o1+i4+Fbfss23Nsz5kyZdj7JiIi4nnoJAxWAatsX1vWL6QJh4fKKR7K85qyfTWwR9vrp5e20dqnD9MeERFdMuYdyLYflHS/pL1s3wkcDNxeHguB08rzReUly4APSDqP5mLxo7YfkLQc+Lu2i8bzgJNtr5P0mKS5wLXAscAXNuPvOKwZJ12ypX9ER+497fDaJUREdDwcxQeBb0jaDrgbOI7mqOJ8SYuA+4Cjyr6XAocBg8CTZV/KH/1PANeX/U61va4svx/4GrAjcFl5REREl3QUBrZvAuYMs+ngYfY1cMII77MEWDJM+wCwdye1RETE5pc7kCMiImEQEREv4CGsY/PJxfSIyJFBREQkDCIiImEQEREkDCIigoRBRESQMIiICBIGERFBwiAiIkgYREQECYOIiCBhEBERJAwiIoKEQUREkDCIiAgSBhERQcIgIiJIGEREBAmDiIggYRARESQMIiKChEFERJAwiIgIEgYREUGHYSDpXkm3SLpJ0kBp21XSCkl3lefJpV2SzpA0KOlmSfu1vc/Csv9dkha2te9f3n+wvFab+xeNiIiRjefI4D/Y3sf2nLJ+EnCF7VnAFWUd4FBgVnksBs6EJjyAU4CDgAOBU1oBUvY5vu1185/3bxQREeO2KaeJFgBLy/JS4Ii29nPduAaYJGl34BBghe11ttcDK4D5ZdvOtq+xbeDctveKiIgu6DQMDFwu6QZJi0vbVNsPlOUHgalleRpwf9trV5W20dpXDdP+HJIWSxqQNLB27doOS4+IiLFM7HC/N9peLekPgBWSft6+0bYlefOXtzHbZwFnAcyZM2eL/7yIiH7R0ZGB7dXleQ3wXZpz/g+VUzyU5zVl99XAHm0vn17aRmufPkx7RER0yZhhIGknSS9pLQPzgFuBZUCrR9BC4KKyvAw4tvQqmgs8Wk4nLQfmSZpcLhzPA5aXbY9Jmlt6ER3b9l4REdEFnZwmmgp8t/T2nAh80/b3JV0PnC9pEXAfcFTZ/1LgMGAQeBI4DsD2OkmfAK4v+51qe11Zfj/wNWBH4LLyiIiILhkzDGzfDbx2mPaHgYOHaTdwwgjvtQRYMkz7ALB3B/VGRMQWkDuQIyIiYRAREQmDiIggYRARESQMIiKChEFERJAwiIgIEgYREUHCICIiSBhERAQJg4iIIGEQEREkDCIigoRBRESQMIiICBIGERFBwiAiIkgYREQECYOIiCBhEBERJAwiIoKEQUREkDCIiAgSBhERQcIgIiJIGEREBOMIA0kTJN0o6eKyPlPStZIGJX1b0nalffuyPli2z2h7j5NL+52SDmlrn1/aBiWdtPl+vYiI6MR4jgxOBO5oWz8d+JztVwDrgUWlfRGwvrR/ruyHpNnA0cCrgfnAl0rATAC+CBwKzAaOKftGRESXdBQGkqYDhwNfLesC3gJcWHZZChxRlheUdcr2g8v+C4DzbP/W9j3AIHBgeQzavtv2U8B5Zd+IiOiSTo8M/hfwl8Dvy/pLgUdsP13WVwHTyvI04H6Asv3Rsv+z7UNeM1L7c0haLGlA0sDatWs7LD0iIsYyZhhIehuwxvYNXahnVLbPsj3H9pwpU6bULiciYqsxsYN93gC8Q9JhwA7AzsDngUmSJpZv/9OB1WX/1cAewCpJE4FdgIfb2lvaXzNSe0REdMGYRwa2T7Y93fYMmgvAP7D9HuBK4F1lt4XARWV5WVmnbP+BbZf2o0tvo5nALOA64HpgVumdtF35Gcs2y28XEREd6eTIYCR/BZwn6ZPAjcDZpf1s4OuSBoF1NH/csX2bpPOB24GngRNsPwMg6QPAcmACsMT2bZtQV0REjNO4wsD2VcBVZflump5AQ/f5DXDkCK//FPCpYdovBS4dTy0REbH55A7kiIhIGERERMIgIiJIGEREBAmDiIggYRARESQMIiKChEFERJAwiIgIEgYREUHCICIiSBhERAQJg4iIIGEQEREkDCIigoRBRESQMIiICBIGERFBwiAiIkgYREQECYOIiCBhEBERJAwiIoKEQUREkDCIiAgSBhERQQdhIGkHSddJ+pmk2yR9vLTPlHStpEFJ35a0XWnfvqwPlu0z2t7r5NJ+p6RD2trnl7ZBSSdt/l8zIiJG08mRwW+Bt9h+LbAPMF/SXOB04HO2XwGsBxaV/RcB60v758p+SJoNHA28GpgPfEnSBEkTgC8ChwKzgWPKvhER0SVjhoEbvy6r25aHgbcAF5b2pcARZXlBWadsP1iSSvt5tn9r+x5gEDiwPAZt3237KeC8sm9ERHRJR9cMyjf4m4A1wArg/wGP2H667LIKmFaWpwH3A5TtjwIvbW8f8pqR2oerY7GkAUkDa9eu7aT0iIjoQEdhYPsZ2/sA02m+yb9qi1Y1ch1n2Z5je86UKVNqlBARsVUaV28i248AVwKvAyZJmlg2TQdWl+XVwB4AZfsuwMPt7UNeM1J7RER0SSe9iaZImlSWdwTeCtxBEwrvKrstBC4qy8vKOmX7D2y7tB9dehvNBGYB1wHXA7NK76TtaC4yL9scv1xERHRm4ti7sDuwtPT62QY43/bFkm4HzpP0SeBG4Oyy/9nA1yUNAuto/rhj+zZJ5wO3A08DJ9h+BkDSB4DlwARgie3bNttvGBERYxozDGzfDOw7TPvdNNcPhrb/BjhyhPf6FPCpYdovBS7toN6IiNgCcgdyREQkDCIiImEQEREkDCIigs56E0X0jRknXVK7BADuPe3w2iVEn8mRQUREJAwiIiJhEBERJAwiIoKEQUREkDCIiAgSBhERQcIgIiJIGEREBAmDiIggYRARESQMIiKChEFERJAwiIgIEgYREUHCICIiSBhERAQJg4iIINNeRsQIMgVof8mRQUREJAwiIqKDMJC0h6QrJd0u6TZJJ5b2XSWtkHRXeZ5c2iXpDEmDkm6WtF/bey0s+98laWFb+/6SbimvOUOStsQvGxERw+vkyOBp4KO2ZwNzgRMkzQZOAq6wPQu4oqwDHArMKo/FwJnQhAdwCnAQcCBwSitAyj7Ht71u/qb/ahER0akxw8D2A7ZXluXHgTuAacACYGnZbSlwRFleAJzrxjXAJEm7A4cAK2yvs70eWAHML9t2tn2NbQPntr1XRER0wbiuGUiaAewLXAtMtf1A2fQgMLUsTwPub3vZqtI2WvuqYdqH+/mLJQ1IGli7du14So+IiFF0HAaSXgx8B/iw7cfat5Vv9N7MtT2H7bNsz7E9Z8qUKVv6x0VE9I2OwkDStjRB8A3b/1SaHyqneCjPa0r7amCPtpdPL22jtU8fpj0iIrqkk95EAs4G7rD92bZNy4BWj6CFwEVt7ceWXkVzgUfL6aTlwDxJk8uF43nA8rLtMUlzy886tu29IiKiCzq5A/kNwHuBWyTdVNr+B3AacL6kRcB9wFFl26XAYcAg8CRwHIDtdZI+AVxf9jvV9rqy/H7ga8COwGXlERERXTJmGNj+ETBSv/+Dh9nfwAkjvNcSYMkw7QPA3mPVEhERW0buQI6IiIRBREQkDCIigoRBRESQMIiICBIGERFBwiAiIkgYREQECYOIiCBhEBERJAwiIoKEQUREkDCIiAgSBhERQcIgIiJIGEREBJ3NdBYR0ddmnHRJ7RIAuPe0w7fYe+fIICIiEgYREZEwiIgIEgYREUHCICIiSBhERAQJg4iIIGEQEREkDCIigg7CQNISSWsk3drWtqukFZLuKs+TS7sknSFpUNLNkvZre83Csv9dkha2te8v6ZbymjMkaXP/khERMbpOjgy+Bswf0nYScIXtWcAVZR3gUGBWeSwGzoQmPIBTgIOAA4FTWgFS9jm+7XVDf1ZERGxhY4aB7auBdUOaFwBLy/JS4Ii29nPduAaYJGl34BBghe11ttcDK4D5ZdvOtq+xbeDctveKiIgueb7XDKbafqAsPwhMLcvTgPvb9ltV2kZrXzVM+7AkLZY0IGlg7dq1z7P0iIgYapMvIJdv9N4MtXTys86yPcf2nClTpnTjR0ZE9IXnGwYPlVM8lOc1pX01sEfbftNL22jt04dpj4iILnq+YbAMaPUIWghc1NZ+bOlVNBd4tJxOWg7MkzS5XDieBywv2x6TNLf0Ijq27b0iIqJLxpzcRtK3gD8GdpO0iqZX0GnA+ZIWAfcBR5XdLwUOAwaBJ4HjAGyvk/QJ4Pqy36m2Wxel30/TY2lH4LLyiIiILhozDGwfM8Kmg4fZ18AJI7zPEmDJMO0DwN5j1REREVtO7kCOiIiEQUREJAwiIoKEQUREkDCIiAgSBhERQcIgIiJIGEREBAmDiIggYRARESQMIiKChEFERJAwiIgIEgYREUHCICIiSBhERAQJg4iIIGEQEREkDCIigoRBRESQMIiICBIGERFBwiAiIkgYREQECYOIiCBhEBER9FAYSJov6U5Jg5JOql1PREQ/6YkwkDQB+CJwKDAbOEbS7LpVRUT0j54IA+BAYND23bafAs4DFlSuKSKib8h27RqQ9C5gvu0/L+vvBQ6y/YEh+y0GFpfVvYA7u1roc+0G/KpyDb0in8UG+Sw2yGexQS98FnvanjLchondrmRT2D4LOKt2HS2SBmzPqV1HL8hnsUE+iw3yWWzQ659Fr5wmWg3s0bY+vbRFREQX9EoYXA/MkjRT0nbA0cCyyjVFRPSNnjhNZPtpSR8AlgMTgCW2b6tcVid65pRVD8hnsUE+iw3yWWzQ059FT1xAjoiIunrlNFFERFSUMIiIiIRBREQkDDaJpG0k7Vy7jl6QzyLihS1hME6SvilpZ0k7AbcCt0v6i9p11ZDPoiHpxPI5SNLZklZKmle7rm6TtOtoj9r1dZukV0q6QtKtZf01kj5Wu66RJAzGb7btx4AjgMuAmcB765ZUTT6LxvvK5zAPmEzzGZxWt6QqbgAGyvNa4F+Au8ryDRXrquUrwMnA7wBs30xzD1VPShiM37aStqX5A7jM9u+Afu2fO9xn0Y9Ung8Dvl7ukdEo+2+VbM+0/UfA/wHebns32y8F3gZcXre6Kl5k+7ohbU9XqaQDCYPx+0fgXmAn4GpJewKPVa2onuE+i0erVlTHDZIupwmD5ZJeAvy+ck01zbV9aWvF9mXA6yvWU8uvJL2c8mWxDMj5QN2SRpabzjYDSRNt92zibymSZtq+p21dwCts31WxrK6TtA2wD3C37UckvRSYVk4L9B1Jy4EfAv+7NL0HeLPtQ+pV1X2S/ojmruPXA+uBe4D32L6vamEjSBh0SNJHRttu+7PdqqVXSFppe78hbTfY3r9WTbVImgbsSdsQL7avrldRPeVi8SnAm0vT1cDHba+rV1X3SZpg+5nSwWIb24/Xrmk0PTE20QvES8rzXsABbBhI7+3A0POCWzVJrwJeDewi6Z1tm3YGdqhTVT2STgfeDdwOPFOaTfNHsO+UP/on1q6jB9wj6fvAt4Ef1C5mLDkyGCdJVwOHt1K+nB++xPabR3/l1kPSApqLxu9g49FlHwfOs/2TKoVVIulO4DW2f1u7ll4g6ZXAfwdmsPGR0ltq1VSDpBfRXDw/GtgPuJjm38ePqhY2goTBOA39hy9pe+Bm23vVraz7JL3O9k9r11GbpMuAI23/unYtvUDSz4Av03QnbR0pYbsfu5cCIGky8HmaawYTatcznJwmGr9zgeskfZem++AC4GtVK+oySV9gQw+JY4Zut/2hrhdV15PATZKuAJ49OujDz6Hladtn1i6iF0j69zSnEOfT3INxVN2KRpYjg+dB0n7Am2j+IP7Q9o2VS+oqSQtH2257abdq6QUjfR799jm0SPpbYA3wXTYOx367gHwvcCNwPs19OE/UrWh0CYPnQdJraXpKtMLgZ5VLisrKDH2vLKt39vENeEi6Z5hmlxvS+oakncud6S8ICYNxknQicDzwHZrTRP8JOMv2F6oWVoGkKxnm7us+vFD4x8BSmhvwRDOf98J+7Vra7yT9pe1Pt59Obderpw9zzWD8FgEHtQ75SrfCnwJ9FwY0PUZadgD+Mz18u/0W9A/APNt3wrO9ab4F9N39FgCSjh2u3fa53a6lkjvK80DVKsYpYTB+oq2HRFnuu3FoYNjeIT+W1Ff3XBTbtoIAwPa/lDGb+tUBbcs7AAcDK2k6X2z1bH+vLD5p+4L2bZKOrFBSRxIG43cOcG3pTQRNf/uzK9ZTzZBhibeh+Sa8S6VyahqQ9FU2Hn7hBfWtcHOy/cH2dUmTgPMqlVPTycAFHbT1hFwzeB4k7Q+8oaz2XW+ilnKh0DRHRk/TjL1yaq/eVLOllHtNTgDeWJp+CHwpN6E1ylHSrf1yL46kQ2kGLTyK5u7jlp1phn0/sEphY0gYPA+SJgBT2fjuyl/Uqyiid0j6HhsunE4A/i1wvu2T6lXVPaW34T7AqcDftG16HLjS9voqhY0hYTBOkj5IMwjXQ2y4XmDbr6laWAXl/Of3bT9eZnDaD/ik7ZWVS+sKSefbPkrSLQzfa6Tv/p+AZ2+0ankauM/2qlr11FKmgX3C9jNlfQKwve0n61Y2vITBOEkapOlN9HDtWmqTdLPt10h6I/BJ4O+Bv7F9UOXSukLS7rYfKPM4PEevDlXcDZKmsuFC8nW219SspwZJ1wD/sTVMiaQXA5fb7sm5HTK5zfjdT39O4DKcVq+qw2nutbgE2K5iPV1luzVRya+A+8sf/+2B1wK/rFZYZZKOohnJ90ia8+bXlold+s0O7eNVleUXVaxnVOlN1KG2+QzuBq6SdAkb32rfd/MZAKsl/SPwVuD0ciG1H79gXA28qQxGdjlwPc14NO+pWlU9fw0c0DoakDSFZirMC6tW1X1PSNqvddq0dDz518o1jShh0LnWfAa/KI/t2PAtuF/PtR1FMwDXZ8oMX7sDf1G5phpk+0lJi2h6EX1a0k21i6pomyGnhR6mP78kfBi4QNIvaa4t/huaLwk9KWHQIdsfh+ai6QvpRpItqfwBXEPTpfIumouFfTXlZSFJr6M5ElhU2npymOIu+X6Z+vJbZf3dwKWj7L9Vsn19mQiq1aW2p8esygXkcRphqsfntPUDSacAc4C9bL9S0suAC2y/YYyXblVK75mPAj+2fXqZ+/bDvToGzZZU5sGeTnPx+Nn7Lmx/d+RXbZ3K5DYfAfa0fbykWTT/Vi6uXNqwEgYdeqHeSLIllVMh+wIrbe9b2m7u1y6V0ZB0i+1/V7uO2iR9m2aCn2Nt713C4Se296lc2rBymqhzv6QZYuAdNP+BWx4H/luViup7yrYltSa62al2QTVk9NbnWCnpANvX1y6kspfbfndrAqhyWrVnxzFLGHSozFnwM0nfpPnc/rB9cLJ+U/6nvrj0Jpok6XjgfcBX6lZWRUZv3dhBwHsk3Qc8Qf/emPmUpB3ZMCvgy2nrgdhrcpponCS9HfgMsJ3tmZL2oRmP5x2VS+u6cuftR4B5NP/gl9teUbeq3iDpun48dQiQm/Aakt4KfAyYTdPl+A3An9m+qmZdI8mRwfj9LXAgcBWA7ZskzaxZUEUrgUds92N30mdl9NbneLzDtq2a7RWSVgJzab4snWj7V5XLGlHCYPx+Z/vRIaf++vXwaujpAKAvx+S5geeO3rpo1Fds3VbSzPa2nuYzmQQ8KOkh4Phh5sHYqkh6le2fl7nSAVp3qv+hpD2Adb14lJQwGL/bJP0JMKF0FfsQ8JPKNdVySO0CeoHtfj0yHMkK4ELbywEkzaO5jnIO8CWaLxFbs48Ai2lmwBvOSyX9zPZ7u1jTmHLNYJxK97C/pjlPDrAc+ETGru8/kt452nbb/9StWnrJcF1L2wY1vKlXu1Z2k6TLbc8be8/uyZHB+M0uj4nlsYCmu2m/nRoJePso2wz0ZRgAD0j6KzbMbvZuYE0Zwvn39crqLkk7AO+nufnONJMefdn2b3otCCBHBuMm6U6aroS30vY/di+eA4yoQdJuNHN+tP4I/phmopdHabpkD1Ysr2sknU9z4bw1HeqfAJNs9+TwNQmDcZL0I9tvHHvP6BeS/g74tO1Hyvpk4KO2P1a3sjokzbR9z5C2vrsJTdLttmeP1dYr+nEkwU11iqSvSjpG0jtbj9pFRVWHtoIAoExreFjFemr7jqRprRVJbwaWVKynlpWS5rZWJB1EM4pBT8o1g/E7DngVsC0bThP18/nhaHqWbd/qRFDuOt2+ck01/Rfgn8sNmvsB/5M+Cse2aVC3BX4i6RdlfU/g5zVrG03CYPwOsL3X2LtFH/kGcIWkc8r6ccDSivVUVYZu/hDNXbe/oZn6cW3lsrrpbW3Lk4E3leWrgUeeu3tvyDWDcSr/4P/e9u21a4neUUa1Pbisrmj1se8nkr7Hxjdgzqa54Wo9QL8N2SLpRODPac4aCDgC+IrtL1QtbAQJg3GSdAfwcpq7TH9L/w7CFbGRMq/DiGz/327V0gsk3Qy8zvYTZX0n4Ke9+rcip4nGb37tAqK3lA4EpwN/QPPloPUFYeeqhXVZ6499GavrAdu/Kes7AlNr1laJgGfa1p8pbT0pYTBOuZ8ghvFp4O2276hdSI+4AHh92/ozpe2AOuVUcw5wraTWLG9HAGdXrGdUCYOITfdQgmAjE20/1Vqx/ZSk7WoWVIPtz0q6ig3Tfx5n+8aKJY0qYRCx6QbKFIf/TNvkJf06NhGwVtI7bC8DkLQA6Nmhm7ck2ytpRnHtebmAHLGJ2rqUtrPt93W9mB5QZvT6BvAymnPk99PMA9wXw1C8UCUMImKLkPRiANu/rl1LjC1hELGJJE0HvkAzrSE0o1OeaHtVvarqknQ48GqaOaEBsH1qvYpiLBmbKGLTnQMsozkt8jLge6WtL0n6Ms2w1R+kOU10JM1QDNHDcmQQsYmGm7ClnydxaZvIpvX8YuAy228a88VRTY4MIjbdw5L+VNKE8vhT4OHaRVX0r+X5SUkvA34H7F6xnuhAwiBi070POAp4kGYsnncBf1azoMouljSJ5ma8G4B7gW9VrSjGlNNEEZtI0lLgw2UeAyTtCnymj7uW7gj8V5rROlvTPZ7ZGp4ielPCIGITSbrR9r5jtfWLEaZ73MX2UfWqirHkDuSITbeNpMlDjgz6+d/W3kOmdrxSUoZ873H9/D9sxObyD8BPJV1Q1o8EPlWxntpWSppr+xro/ekeo5HTRBGbgaTZwFvK6g/6cfKjIdM97gVsNN1jr04EH42EQURsFpJGvbEsw7/3toRBRETkPoOIiEgYREQECYOIiCBhEBERwP8HfcGcxatCXh8AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KHZiL1dwwevI"
      },
      "source": [
        "### Encode data\n",
        "\n",
        " - save text (x) and labels (y) to separate lists to pass on to ML model later on\n",
        " -convet labels to numeric form using OneHotEncoding"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_Ot8KNVhwaqF",
        "outputId": "ac5a17fd-17b7-4af1-cfa6-98fd99e2f2fb"
      },
      "source": [
        "X_train = train_df[\"text\"].to_list()\n",
        "X_test = test_df[\"text\"].to_list()\n",
        "X_val = val_df[\"text\"].to_list()\n",
        "len(X_train), len(X_test), len(X_val)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(180040, 30135, 30212)"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ynK4MpVGwdRJ",
        "outputId": "a9d7a83d-78d9-4282-8592-5dd616f853c9"
      },
      "source": [
        "y_train = train_df[\"label\"].to_list()\n",
        "y_test = test_df[\"label\"].to_list()\n",
        "y_val = val_df[\"label\"].to_list()\n",
        "len(y_train), len(y_test), len(y_val)"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(180040, 30135, 30212)"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "z14BFSeNxRm9",
        "outputId": "4b001498-1ef7-4dc3-85c5-778d631c65c4"
      },
      "source": [
        "X_train[:10]"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['to investigate the efficacy of @ weeks of daily low-dose oral prednisolone in improving pain , mobility , and systemic low-grade inflammation in the short term and whether the effect would be sustained at @ weeks in older adults with moderate to severe knee osteoarthritis ( oa ) .',\n",
              " 'a total of @ patients with primary knee oa were randomized @:@ ; @ received @ mg/day of prednisolone and @ received placebo for @ weeks .',\n",
              " 'outcome measures included pain reduction and improvement in function scores and systemic inflammation markers .',\n",
              " 'pain was assessed using the visual analog pain scale ( @-@ mm ) .',\n",
              " 'secondary outcome measures included the western ontario and mcmaster universities osteoarthritis index scores , patient global assessment ( pga ) of the severity of knee oa , and @-min walk distance ( @mwd ) .',\n",
              " 'serum levels of interleukin @ ( il-@ ) , il-@ , tumor necrosis factor ( tnf ) - , and high-sensitivity c-reactive protein ( hscrp ) were measured .',\n",
              " 'there was a clinically relevant reduction in the intervention group compared to the placebo group for knee pain , physical function , pga , and @mwd at @ weeks .',\n",
              " 'the mean difference between treatment arms ( @ % ci ) was @ ( @-@ @ ) , p < @ ; @ ( @-@ @ ) , p < @ ; @ ( @-@ @ ) , p < @ ; and @ ( @-@ @ ) , p < @ , respectively .',\n",
              " 'further , there was a clinically relevant reduction in the serum levels of il-@ , il-@ , tnf - , and hscrp at @ weeks in the intervention group when compared to the placebo group .',\n",
              " 'these differences remained significant at @ weeks .']"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ATTMS7mJyYqN",
        "outputId": "539c632b-7508-499e-8ae4-0eda87538339"
      },
      "source": [
        "y_train[:5]"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['objective', 'methods', 'methods', 'methods', 'methods']"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "al7t0hA1zNZ1"
      },
      "source": [
        "#get indeces -  convert to array to be ableto use with sklearn OneHotEncoder\n",
        "labels = np.array(y_train)"
      ],
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ll7-a4JB0vBZ",
        "outputId": "bc53f3d4-8121-4d67-b7cd-7f6353a7e1b5"
      },
      "source": [
        "labels"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array(['objective', 'methods', 'methods', ..., 'results', 'conclusions',\n",
              "       'conclusions'], dtype='<U11')"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AkumoYv22zlE"
      },
      "source": [
        "##### One-hot Encoding"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xCEAtEMZyZl9"
      },
      "source": [
        "#one hot encode labels - non-sparse matrix in order to be compatible with tf.constant()\n",
        "enc = OneHotEncoder(sparse=False)"
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c4UEL0ayz5f7"
      },
      "source": [
        "encoded_train_labels = enc.fit_transform(labels.reshape(-1, 1)) #fit to training labels AND trasnform those labels"
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ukHvkXiq1XiY",
        "outputId": "57de3053-e893-4153-c625-6d68de0e8fbd"
      },
      "source": [
        "encoded_train_labels"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0., 0., 0., 1., 0.],\n",
              "       [0., 0., 1., 0., 0.],\n",
              "       [0., 0., 1., 0., 0.],\n",
              "       ...,\n",
              "       [0., 0., 0., 0., 1.],\n",
              "       [0., 1., 0., 0., 0.],\n",
              "       [0., 1., 0., 0., 0.]])"
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "14ntnNExz9DN",
        "outputId": "f15f66ef-9f5a-4007-c5d8-bdaa35332b3e"
      },
      "source": [
        "tf.constant(encoded_train_labels)"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(180040, 5), dtype=float64, numpy=\n",
              "array([[0., 0., 0., 1., 0.],\n",
              "       [0., 0., 1., 0., 0.],\n",
              "       [0., 0., 1., 0., 0.],\n",
              "       ...,\n",
              "       [0., 0., 0., 0., 1.],\n",
              "       [0., 1., 0., 0., 0.],\n",
              "       [0., 1., 0., 0., 0.]])>"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iqSC0IZg1JPO"
      },
      "source": [
        "#use fit_transformed encoder to also trasnform validation and test labels\n",
        "encoded_test_labels = enc.transform(np.array(y_test).reshape(-1,1)) #encoder was already fitted (to training labels) - transform test and val labels now\n",
        "encoded_val_labels = enc.transform(np.array(y_val).reshape(-1,1)) "
      ],
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8zJIe_6P25vN"
      },
      "source": [
        "##### Label Encoding"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mxQS1XPM28Jv",
        "outputId": "8b6bd844-a697-4672-ce13-d33b364096d4"
      },
      "source": [
        "label_enc = LabelEncoder()\n",
        "lab_encoded_train_labels = label_enc.fit_transform(labels.reshape(-1, 1))\n",
        "lab_encoded_test_labels = label_enc.transform(np.array(y_test).reshape(-1,1)) #encoder was already fitted (to training labels) - transform test and val labels now\n",
        "lab_encoded_val_labels = label_enc.transform(np.array(y_val).reshape(-1,1)) "
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/sklearn/preprocessing/_label.py:251: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
            "  y = column_or_1d(y, warn=True)\n",
            "/usr/local/lib/python3.7/dist-packages/sklearn/preprocessing/_label.py:268: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
            "  y = column_or_1d(y, warn=True)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CLgfanAmGsEJ",
        "outputId": "f386ef60-6664-47a4-ac3a-a837d3ba97ed"
      },
      "source": [
        "lab_encoded_train_labels[:10]"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([3, 2, 2, 2, 2, 2, 4, 4, 4, 4])"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EmJKxmE73B41"
      },
      "source": [
        "##### Get class names from encoder\n",
        "\n",
        "--> this could be done with any of the two encoders, but here the information in extracted from OneHotEncoder instance (enc)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5kmHdgKA12Ry",
        "outputId": "26ec7191-f542-4bce-c35a-12c88caa108a"
      },
      "source": [
        "#get class names and numbers from label encoder\n",
        "num_classes = len(label_enc.classes_)\n",
        "classes = list(label_enc.classes_)\n",
        "classes"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['background', 'conclusions', 'methods', 'objective', 'results']"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5mK4Dv02RdVk"
      },
      "source": [
        "# ##for labels from onehotencoder: use\n",
        "# enc.categories_"
      ],
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GR532gSKSy46",
        "outputId": "c6b260b6-483a-401b-e74d-6bb385940e3e"
      },
      "source": [
        "dir(enc.categories_)"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['__add__',\n",
              " '__class__',\n",
              " '__contains__',\n",
              " '__delattr__',\n",
              " '__delitem__',\n",
              " '__dir__',\n",
              " '__doc__',\n",
              " '__eq__',\n",
              " '__format__',\n",
              " '__ge__',\n",
              " '__getattribute__',\n",
              " '__getitem__',\n",
              " '__gt__',\n",
              " '__hash__',\n",
              " '__iadd__',\n",
              " '__imul__',\n",
              " '__init__',\n",
              " '__init_subclass__',\n",
              " '__iter__',\n",
              " '__le__',\n",
              " '__len__',\n",
              " '__lt__',\n",
              " '__mul__',\n",
              " '__ne__',\n",
              " '__new__',\n",
              " '__reduce__',\n",
              " '__reduce_ex__',\n",
              " '__repr__',\n",
              " '__reversed__',\n",
              " '__rmul__',\n",
              " '__setattr__',\n",
              " '__setitem__',\n",
              " '__sizeof__',\n",
              " '__str__',\n",
              " '__subclasshook__',\n",
              " 'append',\n",
              " 'clear',\n",
              " 'copy',\n",
              " 'count',\n",
              " 'extend',\n",
              " 'index',\n",
              " 'insert',\n",
              " 'pop',\n",
              " 'remove',\n",
              " 'reverse',\n",
              " 'sort']"
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UlQYcuPgG-VY"
      },
      "source": [
        "## Train Models\n",
        "\n",
        "Different models will be tried, including:<br>\n",
        " - sklearn's NaiveBayes (tf-idf) encoder <br>\n",
        " - TensorFlow Hub Pretrained Feature Extractor<br>\n",
        " - Conv1D with character embeddings<br>\n",
        " - Pretrained token embeddings (same as 2nd model from above) + charater embeddings (same as 3rd from above)<br>\n",
        " - Pretrained token embeddings (same as 2nd model from above) + charater embeddings (same as 3rd from above) + positional embeddings\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6_nMwE6AHvlw"
      },
      "source": [
        "### Naive Bayes (tf-idf)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6jTGd6XX2pcP"
      },
      "source": [
        "# create model (here: using Pipeline)\n",
        "\n",
        "baseline_model = Pipeline(\n",
        "    [\n",
        "     (\"tf-idf\", TfidfVectorizer()),\n",
        "     (\"clf\", MultinomialNB())\n",
        "    ]\n",
        ")\n"
      ],
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8t_H2S6r2dMB"
      },
      "source": [
        "#compile model not needed for sklearn models"
      ],
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5uQes1VUIB_R",
        "outputId": "5e1f9b65-f5a3-454a-dfa5-f94acf00fbd5"
      },
      "source": [
        "#fit model\n",
        "baseline_model.fit(X=X_train,\n",
        "                   y=lab_encoded_train_labels #using Label Encoded labels\n",
        "                   )"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Pipeline(memory=None,\n",
              "         steps=[('tf-idf',\n",
              "                 TfidfVectorizer(analyzer='word', binary=False,\n",
              "                                 decode_error='strict',\n",
              "                                 dtype=<class 'numpy.float64'>,\n",
              "                                 encoding='utf-8', input='content',\n",
              "                                 lowercase=True, max_df=1.0, max_features=None,\n",
              "                                 min_df=1, ngram_range=(1, 1), norm='l2',\n",
              "                                 preprocessor=None, smooth_idf=True,\n",
              "                                 stop_words=None, strip_accents=None,\n",
              "                                 sublinear_tf=False,\n",
              "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
              "                                 tokenizer=None, use_idf=True,\n",
              "                                 vocabulary=None)),\n",
              "                ('clf',\n",
              "                 MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True))],\n",
              "         verbose=False)"
            ]
          },
          "metadata": {},
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CKcZerCyJFYQ",
        "outputId": "999c7d94-1a9f-45ee-d27c-c5e5e10b0b14"
      },
      "source": [
        "#evaluate baseline model -->NOTE: similar to '.evaluate' in keras\n",
        "baseline_model.score(X=X_val,\n",
        "                        y=lab_encoded_val_labels)"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.7218323844829869"
            ]
          },
          "metadata": {},
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1Tp0G7QpJbdw"
      },
      "source": [
        "#make and save predictions using baseline model\n",
        "baseline_preds = baseline_model.predict(X_val)"
      ],
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YUfbE43TJvQF",
        "outputId": "29235f7f-ea22-4d95-97f4-0320b9ee8efa"
      },
      "source": [
        "baseline_preds"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([4, 1, 3, ..., 4, 4, 1])"
            ]
          },
          "metadata": {},
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-m5TuKsYJwIP"
      },
      "source": [
        "baseline_metrics = calculate_results(lab_encoded_val_labels, baseline_preds)"
      ],
      "execution_count": 51,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l1EpiTW4LL6M"
      },
      "source": [
        "### 1D CNN with Token Embeddings"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LsMqgtwxOGQs",
        "outputId": "f81bea45-859d-4967-b3ab-2b7a74a07b8c"
      },
      "source": [
        "#info regarding sentence length\n",
        "train_df['sentence_length'] = train_df.text.apply(lambda x: len(x.split()))\n",
        "train_df.sentence_length.describe()"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "count    180040.000000\n",
              "mean         26.338269\n",
              "std          15.386803\n",
              "min           1.000000\n",
              "25%          16.000000\n",
              "50%          23.000000\n",
              "75%          33.000000\n",
              "max         296.000000\n",
              "Name: sentence_length, dtype: float64"
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "PGASMhaYY82q",
        "outputId": "80b31a77-85d6-4b37-f024-4d003c419493"
      },
      "source": [
        "plt.hist(train_df.sentence_length, bins = 25)\n",
        "plt.show()"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD4CAYAAAAO9oqkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAU8ElEQVR4nO3df4xd5X3n8fcndiAoKbEJsxayrTXZWo0o2hAzAkeNom5QjA1VzUopIlqtLWTh1QKrRNrV1myldQvNiqy0zdarlMpbvNhRNoTSRlgN1J11iKr9w+ChEH6WekJA2DJ4ivmRFDVZ0u/+cZ9Jboc7M3fMeH6+X9LVfc73POfc5+EYf+b8mOtUFZKkpe19cz0ASdLcMwwkSYaBJMkwkCRhGEiSgOVzPYAzdeGFF9a6devmehiStGA89thjf1tVA73WLdgwWLduHcPDw3M9DElaMJK8NNE6LxNJkgwDSZJhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJIkF/BvIs2ndrm9Pq/+Ld157lkYiSWfHlGcGSX4pyRNdr7eSfDHJBUmGkhxr7ytb/yTZk2QkyZNJNnTta3vrfyzJ9q765UmeatvsSZKzM11JUi9ThkFVPV9Vl1XVZcDlwNvAt4BdwOGqWg8cbssAW4D17bUTuAsgyQXAbuBK4Apg91iAtD43dW23eUZmJ0nqy3TvGVwFfL+qXgK2AvtbfT9wXWtvBQ5UxxFgRZKLgKuBoao6XVWvA0PA5rbu/Ko6Up1/kPlA174kSbNgumFwA/CN1l5VVSdb+xVgVWuvBl7u2uZ4q01WP96j/i5JdiYZTjI8Ojo6zaFLkibSdxgkOQf4deCPx69rP9HXDI6rp6raW1WDVTU4MNDzK7klSWdgOmcGW4C/qqpX2/Kr7RIP7f1Uq58A1nZtt6bVJquv6VGXJM2S6YTB5/n5JSKAg8DYE0HbgQe66tvaU0UbgTfb5aRDwKYkK9uN403AobburSQb21NE27r2JUmaBX39nkGSDwKfBf5NV/lO4L4kO4CXgOtb/UHgGmCEzpNHNwJU1ekkdwBHW7/bq+p0a98M3AOcBzzUXpKkWdJXGFTV3wEfGVd7jc7TReP7FnDLBPvZB+zrUR8GLu1nLJKkmefXUUiSDANJkmEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJNFnGCRZkeT+JH+d5Lkkn0xyQZKhJMfa+8rWN0n2JBlJ8mSSDV372d76H0uyvat+eZKn2jZ7kmTmpypJmki/Zwa/D/x5VX0M+DjwHLALOFxV64HDbRlgC7C+vXYCdwEkuQDYDVwJXAHsHguQ1uemru02v7dpSZKmY8owSPJh4NPA3QBV9ZOqegPYCuxv3fYD17X2VuBAdRwBViS5CLgaGKqq01X1OjAEbG7rzq+qI1VVwIGufUmSZkE/ZwYXA6PA/0ryeJI/SvJBYFVVnWx9XgFWtfZq4OWu7Y+32mT14z3q75JkZ5LhJMOjo6N9DF2S1I9+wmA5sAG4q6o+AfwdP78kBED7ib5mfnj/WFXtrarBqhocGBg42x8nSUtGP2FwHDheVY+05fvphMOr7RIP7f1UW38CWNu1/ZpWm6y+pkddkjRLpgyDqnoFeDnJL7XSVcCzwEFg7Img7cADrX0Q2NaeKtoIvNkuJx0CNiVZ2W4cbwIOtXVvJdnYniLa1rUvSdIsWN5nv38HfD3JOcALwI10guS+JDuAl4DrW98HgWuAEeDt1peqOp3kDuBo63d7VZ1u7ZuBe4DzgIfaS5I0S/oKg6p6AhjsseqqHn0LuGWC/ewD9vWoDwOX9jMWSdLM8zeQJUmGgSTJMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiS6DMMkryY5KkkTyQZbrULkgwlOdbeV7Z6kuxJMpLkySQbuvazvfU/lmR7V/3ytv+Rtm1meqKSpIlN58zgX1TVZVU12JZ3AYeraj1wuC0DbAHWt9dO4C7ohAewG7gSuALYPRYgrc9NXdttPuMZSZKm7b1cJtoK7G/t/cB1XfUD1XEEWJHkIuBqYKiqTlfV68AQsLmtO7+qjlRVAQe69iVJmgX9hkEBf5HksSQ7W21VVZ1s7VeAVa29Gni5a9vjrTZZ/XiP+rsk2ZlkOMnw6Ohon0OXJE1leZ/9PlVVJ5L8E2AoyV93r6yqSlIzP7x/rKr2AnsBBgcHz/rnSdJS0deZQVWdaO+ngG/Rueb/arvEQ3s/1bqfANZ2bb6m1Sarr+lRlyTNkinPDJJ8EHhfVf2wtTcBtwMHge3Ane39gbbJQeDWJPfSuVn8ZlWdTHII+C9dN403AbdV1ekkbyXZCDwCbAP+x8xNcfat2/XtafV/8c5rz9JIJKk//VwmWgV8qz3tuRz431X150mOAvcl2QG8BFzf+j8IXAOMAG8DNwK0v/TvAI62frdX1enWvhm4BzgPeKi9JEmzZMowqKoXgI/3qL8GXNWjXsAtE+xrH7CvR30YuLSP8UqSzgJ/A1mSZBhIkgwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJIlphEGSZUkeT/JnbfniJI8kGUnyzSTntPq5bXmkrV/XtY/bWv35JFd31Te32kiSXTM3PUlSP6ZzZvAF4Lmu5S8DX6mqXwReB3a0+g7g9Vb/SutHkkuAG4BfBjYDf9ACZhnwVWALcAnw+dZXkjRL+gqDJGuAa4E/assBPgPc37rsB65r7a1tmbb+qtZ/K3BvVf24qn4AjABXtNdIVb1QVT8B7m19JUmzpN8zg/8O/EfgH9ryR4A3quqdtnwcWN3aq4GXAdr6N1v/n9XHbTNR/V2S7EwynGR4dHS0z6FLkqYyZRgk+TXgVFU9NgvjmVRV7a2qwaoaHBgYmOvhSNKisbyPPr8C/HqSa4APAOcDvw+sSLK8/fS/BjjR+p8A1gLHkywHPgy81lUf073NRHVJ0iyY8sygqm6rqjVVtY7ODeDvVNW/Ah4GPte6bQceaO2DbZm2/jtVVa1+Q3va6GJgPfAocBRY355OOqd9xsEZmZ0kqS/9nBlM5DeBe5P8LvA4cHer3w18LckIcJrOX+5U1TNJ7gOeBd4BbqmqnwIkuRU4BCwD9lXVM+9hXJKkaZpWGFTVd4HvtvYLdJ4EGt/n74HfmGD7LwFf6lF/EHhwOmORJM0cfwNZkmQYSJIMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSSJPsIgyQeSPJrke0meSfI7rX5xkkeSjCT5ZpJzWv3ctjzS1q/r2tdtrf58kqu76ptbbSTJrpmfpiRpMv2cGfwY+ExVfRy4DNicZCPwZeArVfWLwOvAjtZ/B/B6q3+l9SPJJcANwC8Dm4E/SLIsyTLgq8AW4BLg862vJGmWTBkG1fGjtvj+9irgM8D9rb4fuK61t7Zl2vqrkqTV762qH1fVD4AR4Ir2GqmqF6rqJ8C9ra8kaZb0dc+g/QT/BHAKGAK+D7xRVe+0LseB1a29GngZoK1/E/hId33cNhPVe41jZ5LhJMOjo6P9DF2S1Ie+wqCqflpVlwFr6Pwk/7GzOqqJx7G3qgaranBgYGAuhiBJi9K0niaqqjeAh4FPAiuSLG+r1gAnWvsEsBagrf8w8Fp3fdw2E9UlSbOkn6eJBpKsaO3zgM8Cz9EJhc+1btuBB1r7YFumrf9OVVWr39CeNroYWA88ChwF1renk86hc5P54ExMTpLUn+VTd+EiYH976ud9wH1V9WdJngXuTfK7wOPA3a3/3cDXkowAp+n85U5VPZPkPuBZ4B3glqr6KUCSW4FDwDJgX1U9M2MzlCRNacowqKongU/0qL9A5/7B+PrfA78xwb6+BHypR/1B4ME+xitJOgv8DWRJkmEgSTIMJEkYBpIkDANJEoaBJAnDQJJEf790tuis2/XtuR6CJM0rnhlIkgwDSZJhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJNFHGCRZm+ThJM8meSbJF1r9giRDSY6195WtniR7kowkeTLJhq59bW/9jyXZ3lW/PMlTbZs9SXI2JitJ6q2fM4N3gH9fVZcAG4FbklwC7AIOV9V64HBbBtgCrG+vncBd0AkPYDdwJXAFsHssQFqfm7q22/zepyZJ6teUX2FdVSeBk639wyTPAauBrcCvtm77ge8Cv9nqB6qqgCNJViS5qPUdqqrTAEmGgM1JvgucX1VHWv0AcB3w0MxMcf6b7ldqv3jntWdpJJKWqmndM0iyDvgE8AiwqgUFwCvAqtZeDbzctdnxVpusfrxHvdfn70wynGR4dHR0OkOXJE2i7zBI8iHgT4AvVtVb3evaWUDN8Njepar2VtVgVQ0ODAyc7Y+TpCWjrzBI8n46QfD1qvrTVn61Xf6hvZ9q9RPA2q7N17TaZPU1PeqSpFnSz9NEAe4Gnquq3+tadRAYeyJoO/BAV31be6poI/Bmu5x0CNiUZGW7cbwJONTWvZVkY/usbV37kiTNgn7+DeRfAf418FSSJ1rtPwF3Avcl2QG8BFzf1j0IXAOMAG8DNwJU1ekkdwBHW7/bx24mAzcD9wDn0blxvGRuHkvSfNDP00T/F5jouf+revQv4JYJ9rUP2NejPgxcOtVYJElnh7+BLEkyDCRJhoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CSRB9hkGRfklNJnu6qXZBkKMmx9r6y1ZNkT5KRJE8m2dC1zfbW/1iS7V31y5M81bbZkyQzPUlJ0uT6OTO4B9g8rrYLOFxV64HDbRlgC7C+vXYCd0EnPIDdwJXAFcDusQBpfW7q2m78Z0mSzrIpw6Cq/hI4Pa68Fdjf2vuB67rqB6rjCLAiyUXA1cBQVZ2uqteBIWBzW3d+VR2pqgIOdO1LkjRLzvSewaqqOtnarwCrWns18HJXv+OtNln9eI96T0l2JhlOMjw6OnqGQ5ckjfeebyC3n+hrBsbSz2ftrarBqhocGBiYjY+UpCXhTMPg1XaJh/Z+qtVPAGu7+q1ptcnqa3rUJUmz6EzD4CAw9kTQduCBrvq29lTRRuDNdjnpELApycp243gTcKiteyvJxvYU0baufUmSZsnyqTok+Qbwq8CFSY7TeSroTuC+JDuAl4DrW/cHgWuAEeBt4EaAqjqd5A7gaOt3e1WN3ZS+mc4TS+cBD7WXJGkWTRkGVfX5CVZd1aNvAbdMsJ99wL4e9WHg0qnGIUk6e/wNZEnS1GcGmn/W7fr2tLd58c5rz8JIJC0WnhlIkgwDSZJhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwu8mWjKm+31GfpeRtLR4ZiBJMgwkSYaBJAnDQJKEN5A1AW84S0uLZwaSpPkTBkk2J3k+yUiSXXM9HklaSubFZaIky4CvAp8FjgNHkxysqmfndmTql5eVpIVtXoQBcAUwUlUvACS5F9gKGAaL1HTD40wYOFL/5ksYrAZe7lo+Dlw5vlOSncDOtvijJM+fwWddCPztGWw3Hy2mucAMzydfnqk9nbHFdHwW01xgcc1nOnP5pxOtmC9h0Jeq2gvsfS/7SDJcVYMzNKQ5tZjmAs5nPltMc4HFNZ+Zmst8uYF8Aljbtbym1SRJs2C+hMFRYH2Si5OcA9wAHJzjMUnSkjEvLhNV1TtJbgUOAcuAfVX1zFn6uPd0mWmeWUxzAeczny2mucDims+MzCVVNRP7kSQtYPPlMpEkaQ4ZBpKkpRMGi+HrLpK8mOSpJE8kGW61C5IMJTnW3lfO9TgnkmRfklNJnu6q9Rx/Ova04/Vkkg1zN/J3m2Auv53kRDs+TyS5pmvdbW0uzye5em5G3VuStUkeTvJskmeSfKHVF+qxmWg+C/X4fCDJo0m+1+bzO61+cZJH2ri/2R6+Icm5bXmkrV/X1wdV1aJ/0bkp/X3go8A5wPeAS+Z6XGcwjxeBC8fV/iuwq7V3AV+e63FOMv5PAxuAp6caP3AN8BAQYCPwyFyPv4+5/DbwH3r0vaT9mTsXuLj9WVw213PoGt9FwIbW/gXgb9qYF+qxmWg+C/X4BPhQa78feKT9d78PuKHV/xD4t619M/CHrX0D8M1+PmepnBn87OsuquonwNjXXSwGW4H9rb0fuG4OxzKpqvpL4PS48kTj3wocqI4jwIokF83OSKc2wVwmshW4t6p+XFU/AEbo/JmcF6rqZFX9VWv/EHiOzrcCLNRjM9F8JjLfj09V1Y/a4vvbq4DPAPe3+vjjM3bc7geuSpKpPmephEGvr7uY7A/HfFXAXyR5rH01B8CqqjrZ2q8Aq+ZmaGdsovEv1GN2a7t0sq/rkt2CmUu7pPAJOj99LvhjM24+sECPT5JlSZ4ATgFDdM5e3qiqd1qX7jH/bD5t/ZvAR6b6jKUSBovFp6pqA7AFuCXJp7tXVue8cME+K7zQxw/cBfwz4DLgJPDf5nY405PkQ8CfAF+sqre61y3EY9NjPgv2+FTVT6vqMjrfznAF8LGZ/oylEgaL4usuqupEez8FfIvOH4pXx07R2/upuRvhGZlo/AvumFXVq+1/2n8A/ic/v9Qw7+eS5P10/uL8elX9aSsv2GPTaz4L+fiMqao3gIeBT9K5PDf2i8PdY/7ZfNr6DwOvTbXvpRIGC/7rLpJ8MMkvjLWBTcDTdOaxvXXbDjwwNyM8YxON/yCwrT25shF4s+uSxbw07rr5v6RzfKAzlxvaUx4XA+uBR2d7fBNp15PvBp6rqt/rWrUgj81E81nAx2cgyYrWPo/Ov/vyHJ1Q+FzrNv74jB23zwHfaWd2k5vrO+Wz9aLzBMTf0LnW9ltzPZ4zGP9H6Tzx8D3gmbE50LkWeBg4Bvwf4IK5Huskc/gGndPz/0fnGueOicZP5wmKr7bj9RQwONfj72MuX2tjfbL9D3lRV//fanN5Htgy1+MfN5dP0bkE9CTwRHtds4CPzUTzWajH558Dj7dxPw3851b/KJ3QGgH+GDi31T/Qlkfa+o/28zl+HYUkaclcJpIkTcIwkCQZBpIkw0CShGEgScIwkCRhGEiSgP8PQxkL94Jr28MAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O7F5vNhZOROU",
        "outputId": "8712254d-f3b2-4ddc-a128-c57c2a7e775b"
      },
      "source": [
        "#get 75% and 95 % int to use in tokenizer output length\n",
        "txt_len_75 = int(train_df.sentence_length.describe()['75%'])\n",
        "txt_len_95 = int(np.percentile(train_df.sentence_length, 95))\n",
        "txt_len_75, txt_len_95"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(33, 55)"
            ]
          },
          "metadata": {},
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8qceEWZxO3Vg"
      },
      "source": [
        "NOTE: sentence length spread is quite extreme and there is an abundance of values with more than 33 words, 95% of all sentences fall into the lengths of 55 (or under) --> using 50 words as output length to account for most sentences lengths"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CppK-_8hO0Of"
      },
      "source": [
        "out_len = 50\n",
        "MAX_TOKENS = 68000 #number from Table 2 in paper cited above"
      ],
      "execution_count": 55,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yVFDt1KNKdnc"
      },
      "source": [
        "#create tokenizer / vectorizerlayer\n",
        "vectorizer = tf.keras.layers.TextVectorization(\n",
        "    max_tokens=MAX_TOKENS, standardize='lower_and_strip_punctuation',\n",
        "    split='whitespace', ngrams=None, output_mode='int',\n",
        "    output_sequence_length=out_len, pad_to_max_tokens=True, vocabulary=None\n",
        ")\n",
        "#fit tokenizer to train sentences\n",
        "vectorizer.adapt(X_train)"
      ],
      "execution_count": 56,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f_J7EZnyPGKR",
        "outputId": "273c9f13-897f-4195-dcc6-3d2443da97ab"
      },
      "source": [
        "#check tokenizer vocab\n",
        "len(vectorizer.get_vocabulary()), vectorizer.get_vocabulary()[-10:]"
      ],
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(64841,\n",
              " ['aarm',\n",
              "  'aaqol',\n",
              "  'aaq',\n",
              "  'aanhui',\n",
              "  'aana',\n",
              "  'aainduced',\n",
              "  'aaigroup',\n",
              "  'aachener',\n",
              "  'aachen',\n",
              "  'aaacp'])"
            ]
          },
          "metadata": {},
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ti0WREu7PUeS",
        "outputId": "6ad32045-7978-465f-d4a5-4f140f4b8b3d"
      },
      "source": [
        "\"bioinformatics\" in vectorizer.get_vocabulary()"
      ],
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "False"
            ]
          },
          "metadata": {},
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cZNPjE1pKdlP"
      },
      "source": [
        "#embed\n",
        "embedder = tf.keras.layers.Embedding(input_dim=len(vectorizer.get_vocabulary()), # set input shape --> = length of vocabulary\n",
        "                             output_dim=516, # set size of embedding vector,\n",
        "                             mask_zero=True,\n",
        "                             embeddings_initializer=\"uniform\", # default, intialize randomly\n",
        "                             input_length=out_len) # how long is each input "
      ],
      "execution_count": 59,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HBy1ur5Waaow",
        "outputId": "090c2931-1f41-4e8a-9431-084cc8e3c359"
      },
      "source": [
        "random_text = random.choice(X_train)\n",
        "print(random_text)\n",
        "print(vectorizer([random_text]))\n",
        "print(embedder(vectorizer([random_text])))"
      ],
      "execution_count": 60,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "occluded vision in seniors seems to prolong this period up to @ min .\n",
            "tf.Tensor(\n",
            "[[8213 1922    5 8575 1529    6 5487   23  173  303    6  242    0    0\n",
            "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
            "     0    0    0    0    0    0    0    0]], shape=(1, 50), dtype=int64)\n",
            "tf.Tensor(\n",
            "[[[-0.0087521   0.01704646  0.04905221 ...  0.02989708  0.03858671\n",
            "   -0.02399713]\n",
            "  [-0.00540137 -0.02095549 -0.00614965 ...  0.00257479 -0.03523976\n",
            "    0.01282111]\n",
            "  [ 0.02184042 -0.00874195 -0.01212107 ...  0.03159361  0.0317611\n",
            "    0.03523907]\n",
            "  ...\n",
            "  [-0.04561064  0.02651793 -0.00658645 ...  0.03354182  0.01081447\n",
            "   -0.00603552]\n",
            "  [-0.04561064  0.02651793 -0.00658645 ...  0.03354182  0.01081447\n",
            "   -0.00603552]\n",
            "  [-0.04561064  0.02651793 -0.00658645 ...  0.03354182  0.01081447\n",
            "   -0.00603552]]], shape=(1, 50, 516), dtype=float32)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u6LjFZAvcxBG"
      },
      "source": [
        "#convert data into tf datasets, so that running efficiency is increased when loading data into model\n",
        "train_dataset = tf.data.Dataset.from_tensor_slices((X_train, encoded_train_labels))\n",
        "test_dataset = tf.data.Dataset.from_tensor_slices((X_test, encoded_test_labels))\n",
        "val_dataset = tf.data.Dataset.from_tensor_slices((X_val, encoded_val_labels))"
      ],
      "execution_count": 61,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9IMU91rmdNmx",
        "outputId": "b308e3c2-526a-4dc1-83d5-82a4d8aa001c"
      },
      "source": [
        "train_dataset"
      ],
      "execution_count": 62,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<TensorSliceDataset shapes: ((), (5,)), types: (tf.string, tf.float64)>"
            ]
          },
          "metadata": {},
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ypJ4WSJOdfEm"
      },
      "source": [
        "#create dataset batches\n",
        "train_dataset = train_dataset.batch(32).prefetch(tf.data.AUTOTUNE)\n",
        "test_dataset = test_dataset.batch(32).prefetch(tf.data.AUTOTUNE)\n",
        "val_dataset = val_dataset.batch(32).prefetch(tf.data.AUTOTUNE)"
      ],
      "execution_count": 63,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "g1ohuRAQduqN",
        "outputId": "81aaf7fa-6eeb-4ed7-8649-00fe173f97b6"
      },
      "source": [
        "train_dataset"
      ],
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<PrefetchDataset shapes: ((None,), (None, 5)), types: (tf.string, tf.float64)>"
            ]
          },
          "metadata": {},
          "execution_count": 64
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vz0LfhtuKdjo"
      },
      "source": [
        "#build model - using functional keras\n",
        "inputs = tf.keras.layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = vectorizer(inputs)\n",
        "x = embedder(x)\n",
        "x = tf.keras.layers.Conv1D(filters=64, kernel_size=5, padding=\"same\", activation=\"relu\")(x)\n",
        "x = tf.keras.layers.GlobalMaxPool1D()(x)\n",
        "outputs = tf.keras.layers.Dense(num_classes, activation=\"softmax\")(x)  #use SOFTMAX for non-binary categorical!!\n",
        "Conv1D_model = tf.keras.Model(inputs, outputs, name=\"Conv1D\")"
      ],
      "execution_count": 65,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n6ECo2RtWTns",
        "outputId": "1b1af660-9fbb-4fd0-a64f-e1ee64dbdb52"
      },
      "source": [
        "#summarize model\n",
        "Conv1D_model.summary()"
      ],
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"Conv1D\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_1 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization (TextVect (None, 50)                0         \n",
            "_________________________________________________________________\n",
            "embedding (Embedding)        (None, 50, 516)           33457956  \n",
            "_________________________________________________________________\n",
            "conv1d (Conv1D)              (None, 50, 64)            165184    \n",
            "_________________________________________________________________\n",
            "global_max_pooling1d (Global (None, 64)                0         \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 5)                 325       \n",
            "=================================================================\n",
            "Total params: 33,623,465\n",
            "Trainable params: 33,623,465\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2vp0abzfKdg-"
      },
      "source": [
        "#compile\n",
        "Conv1D_model.compile(\n",
        "    loss=\"categorical_crossentropy\",\n",
        "    optimizer=tf.keras.optimizers.Adam(),\n",
        "    metrics=[\"accuracy\"]\n",
        ")"
      ],
      "execution_count": 67,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3W7qYk2nLZIt",
        "outputId": "6160bfbe-d029-4922-fff8-e1238659c351"
      },
      "source": [
        "#fit\n",
        "conv1D_history = Conv1D_model.fit(\n",
        "    train_dataset,\n",
        "    steps_per_epoch=len(train_dataset),\n",
        "    epochs=3,\n",
        "    validation_data=val_dataset,\n",
        "    validation_steps=int(0.1*len(val_dataset))\n",
        ")"
      ],
      "execution_count": 68,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/3\n",
            "5627/5627 [==============================] - 238s 37ms/step - loss: 0.5457 - accuracy: 0.7986 - val_loss: 0.5011 - val_accuracy: 0.8122\n",
            "Epoch 2/3\n",
            "5627/5627 [==============================] - 207s 37ms/step - loss: 0.3360 - accuracy: 0.8821 - val_loss: 0.5806 - val_accuracy: 0.8042\n",
            "Epoch 3/3\n",
            "5627/5627 [==============================] - 207s 37ms/step - loss: 0.1765 - accuracy: 0.9425 - val_loss: 0.7331 - val_accuracy: 0.7889\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AMZBd7xnKde0",
        "outputId": "87f139a4-a1de-481b-8501-86970acc469f"
      },
      "source": [
        "Conv1D_model.evaluate(val_dataset)"
      ],
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "945/945 [==============================] - 6s 6ms/step - loss: 0.7310 - accuracy: 0.7926\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.731009304523468, 0.7926320433616638]"
            ]
          },
          "metadata": {},
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_b_LsTm8DRnR",
        "outputId": "4a3d7896-d4d4-4890-9a70-4d9135a84e0e"
      },
      "source": [
        "!mkdir -p saved_model\n",
        "Conv1D_model.save('saved_model/Conv1D_model1')"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO:tensorflow:Assets written to: saved_model/Conv1D_model1/assets\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E5eB_fkcDjrl"
      },
      "source": [
        "conv1d_preds_probs = Conv1D_model.predict(val_dataset)"
      ],
      "execution_count": 71,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FTgW_wWbD0mB"
      },
      "source": [
        "conv1d_preds = tf.math.argmax(conv1d_preds_probs, axis=1)"
      ],
      "execution_count": 72,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "svQMNE2_D-3g",
        "outputId": "7acb89bd-1c92-481b-c05a-fb0208a5b243"
      },
      "source": [
        "lab_encoded_val_labels"
      ],
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0, 0, 3, ..., 4, 1, 1])"
            ]
          },
          "metadata": {},
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_wDLeWMhEqkG",
        "outputId": "4834a0ab-607f-4c08-fc0c-b8f47e7c036b"
      },
      "source": [
        "conv1D_metrics = calculate_results(lab_encoded_val_labels, conv1d_preds)\n",
        "conv1D_metrics"
      ],
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 79.26320667284523,\n",
              " 'f1': 0.7898031817477734,\n",
              " 'precision': 0.7910569268842946,\n",
              " 'recall': 0.7926320667284523}"
            ]
          },
          "metadata": {},
          "execution_count": 80
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "42Ptoyjvl0dV"
      },
      "source": [
        "# conv1D_metrics = {'accuracy': 78.81967430160202,\n",
        "#  'f1': 0.7859960928563114,\n",
        "#  'precision': 0.7892146068302194,\n",
        "#  'recall': 0.7881967430160202}"
      ],
      "execution_count": 79,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Sq8B0pCWFK0g",
        "outputId": "32875415-86c8-45d8-f3f3-1a33c4b81927"
      },
      "source": [
        "baseline_metrics"
      ],
      "execution_count": 81,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 72.1832384482987,\n",
              " 'f1': 0.6989250353450294,\n",
              " 'precision': 0.7186466952323352,\n",
              " 'recall': 0.7218323844829869}"
            ]
          },
          "metadata": {},
          "execution_count": 81
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZAevqgQ-KeAk"
      },
      "source": [
        "Helper functions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iCsl_uNQKdcE"
      },
      "source": [
        "def calculate_results(y_true, y_pred):\n",
        "  \"\"\"\n",
        "  Calculates model accuracy, precision, recall and f1 score of a binary classification model.\n",
        "  PARAMS:\n",
        "    y_true(np.array) - true labels in the form of a 1D array\n",
        "    y_pred(np.array) - predicted labels in the form of a 1D array\n",
        "  RETURNS:\n",
        "    model_results (dict) - a dictionary of accuracy, precision, recall, f1-score.\n",
        "  \"\"\"\n",
        "  from sklearn.metrics import accuracy_score, precision_recall_fscore_support\n",
        "  # Calculate model accuracy\n",
        "  model_accuracy = accuracy_score(y_true, y_pred) * 100\n",
        "  # Calculate model precision, recall and f1 score using \"weighted average\n",
        "  model_precision, model_recall, model_f1, _ = precision_recall_fscore_support(y_true, y_pred, average=\"weighted\")\n",
        "  model_results = {\"accuracy\": model_accuracy,\n",
        "                  \"precision\": model_precision,\n",
        "                  \"recall\": model_recall,\n",
        "                  \"f1\": model_f1}\n",
        "  return model_results"
      ],
      "execution_count": 82,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "STNfphSvn-Vt"
      },
      "source": [
        "### Transfer Learning Model: with Tensorflow model + Feature Extraction and pretrained token embeddings"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YYQNOBLvl-NU"
      },
      "source": [
        "#load pre-trained model as base for this model; feature extraction pnly, so --> not trainable\n",
        "use_layer = hub.KerasLayer(\"https://tfhub.dev/google/universal-sentence-encoder/4\",\n",
        "                                        input_shape=[], \n",
        "                                        dtype=tf.string, \n",
        "                                        trainable=False, \n",
        "                                        name=\"use\") "
      ],
      "execution_count": 85,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "oyQZXfbCq_es",
        "outputId": "0deaaff1-4f61-491d-c964-59a36257cc7b"
      },
      "source": [
        "random_text"
      ],
      "execution_count": 87,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'occluded vision in seniors seems to prolong this period up to @ min .'"
            ]
          },
          "metadata": {},
          "execution_count": 87
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p_Sd5FUbr5dp",
        "outputId": "d6527c6e-97d8-4fd8-8c51-5eb9475020da"
      },
      "source": [
        "# check if loaded model works + length of output (=a 512-vector)\n",
        "len(use_layer([random_text])[0])"
      ],
      "execution_count": 94,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "512"
            ]
          },
          "metadata": {},
          "execution_count": 94
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ttzFqd9JsJNM"
      },
      "source": [
        "#build model based on pretrained embedding (use)\n",
        "inputs = tf.keras.layers.Input(shape=[], dtype=tf.string)\n",
        "pretrained_embedding = use_layer(inputs)\n",
        "x = tf.keras.layers.Dense(128, activation=\"relu\")(pretrained_embedding)\n",
        "output = tf.keras.layers.Dense(num_classes, activation=\"softmax\")(x)\n",
        "model_pretrained_embedding = tf.keras.Model(inputs, output)"
      ],
      "execution_count": 98,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5GNcJvybt_Y7"
      },
      "source": [
        "#compile model\n",
        "model_pretrained_embedding.compile(\n",
        "    loss=\"categorical_crossentropy\",\n",
        "    optimizer=tf.keras.optimizers.Adam(),\n",
        "    metrics=[\"accuracy\"]\n",
        ")"
      ],
      "execution_count": 100,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7K3sLLWSu41v",
        "outputId": "181e62bf-32e5-40ff-d561-58e543330a0e"
      },
      "source": [
        "model_pretrained_embedding.summary()"
      ],
      "execution_count": 102,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_2 (InputLayer)         [(None,)]                 0         \n",
            "_________________________________________________________________\n",
            "use (KerasLayer)             (None, 512)               256797824 \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 64)                32832     \n",
            "_________________________________________________________________\n",
            "dense_2 (Dense)              (None, 5)                 325       \n",
            "=================================================================\n",
            "Total params: 256,830,981\n",
            "Trainable params: 33,157\n",
            "Non-trainable params: 256,797,824\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dwmVaEXsuWUc",
        "outputId": "9e0684cd-531b-4268-b738-cfc721b87fa2"
      },
      "source": [
        "#fit model\n",
        "model_pretrained_embedding.fit(\n",
        "    train_dataset,\n",
        "    steps_per_epoch=len(train_dataset),\n",
        "    epochs=3,\n",
        "    validation_data=val_dataset,\n",
        "    validation_steps=int(0.1*len(val_dataset))\n",
        ")"
      ],
      "execution_count": 101,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/3\n",
            "5627/5627 [==============================] - 109s 19ms/step - loss: 0.7533 - accuracy: 0.7122 - val_loss: 0.6752 - val_accuracy: 0.7427\n",
            "Epoch 2/3\n",
            "5627/5627 [==============================] - 107s 19ms/step - loss: 0.6611 - accuracy: 0.7487 - val_loss: 0.6401 - val_accuracy: 0.7606\n",
            "Epoch 3/3\n",
            "5627/5627 [==============================] - 107s 19ms/step - loss: 0.6288 - accuracy: 0.7613 - val_loss: 0.6240 - val_accuracy: 0.7630\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f21051bf810>"
            ]
          },
          "metadata": {},
          "execution_count": 101
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ji6iRSzTuzmM",
        "outputId": "1c5bc94b-594e-4cfd-805f-e2226285f1c4"
      },
      "source": [
        "#evaluate model, predict and get performance metrics\n",
        "model_pretrained_embedding.evaluate(val_dataset)"
      ],
      "execution_count": 103,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "945/945 [==============================] - 17s 18ms/step - loss: 0.6270 - accuracy: 0.7601\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.6269599795341492, 0.7600952982902527]"
            ]
          },
          "metadata": {},
          "execution_count": 103
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t3zHe-nIwoV0"
      },
      "source": [
        "model_pretrained_embedding_preds = model_pretrained_embedding.predict(val_dataset)"
      ],
      "execution_count": 113,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w2wV92YSwveQ"
      },
      "source": [
        "model_pretrained_embedding_preds = np.array(tf.math.argmax(model_pretrained_embedding_preds, axis=1))"
      ],
      "execution_count": 114,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ImYhXPuWxLZQ",
        "outputId": "d39f3f50-b547-4b42-f36d-321202aa670b"
      },
      "source": [
        "model_pretrained_embedding_preds"
      ],
      "execution_count": 115,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0, 1, 0, ..., 4, 4, 1])"
            ]
          },
          "metadata": {},
          "execution_count": 115
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_kEjm5OCw5j7",
        "outputId": "038e7164-1a06-47c2-cf63-1be979261287"
      },
      "source": [
        "lab_encoded_val_labels"
      ],
      "execution_count": 116,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0, 0, 3, ..., 4, 1, 1])"
            ]
          },
          "metadata": {},
          "execution_count": 116
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D2FM-MY-xYQU",
        "outputId": "ee18b3f9-fa0c-4daf-ab78-b0d80abdaedd"
      },
      "source": [
        "model_pretrained_embedding_metrics = calculate_results(lab_encoded_val_labels, model_pretrained_embedding_preds)\n",
        "model_pretrained_embedding_metrics"
      ],
      "execution_count": 118,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 76.00953263603866,\n",
              " 'f1': 0.7553966219715307,\n",
              " 'precision': 0.7571118640167971,\n",
              " 'recall': 0.7600953263603866}"
            ]
          },
          "metadata": {},
          "execution_count": 118
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bEBp3mpqxdWJ",
        "outputId": "4eb160e1-0d4e-425a-fb29-3bf874dcb76a"
      },
      "source": [
        "conv1D_metrics"
      ],
      "execution_count": 119,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 79.26320667284523,\n",
              " 'f1': 0.7898031817477734,\n",
              " 'precision': 0.7910569268842946,\n",
              " 'recall': 0.7926320667284523}"
            ]
          },
          "metadata": {},
          "execution_count": 119
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DFBEeK76x5Vz"
      },
      "source": [
        "### Conv1D with character embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xUrDL3H8ykIu"
      },
      "source": [
        "#####  create text vectorizer that works at character-level"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xsvg7PGaxxoo"
      },
      "source": [
        "#convert word-based sentences into string of individual characters\n",
        "def sentence_to_char(sentence_list):\n",
        "  \"\"\"\n",
        "  Takes list of sentences and separates all words in each sentece into individual characters\n",
        "  \"\"\"\n",
        "  char_list = [\" \".join(list(sentence)) for sentence in sentence_list]  #list(string) - converts string into list of individual characters in the string\n",
        "  return char_list"
      ],
      "execution_count": 123,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hQ5WMQ4szkPq"
      },
      "source": [
        "X_train_chars = sentence_to_char(X_train)\n",
        "X_test_chars = sentence_to_char(X_test)\n",
        "X_val_chars = sentence_to_char(X_val)"
      ],
      "execution_count": 127,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2RSrW0Tbz8nm",
        "outputId": "1539aa4b-db6f-4aea-ecd6-9c2ef7f749a6"
      },
      "source": [
        "#calculate average number of characters in training sentences --> see length in cell below; NOTE:output of this cell also counts introduced whitespace\n",
        "np.mean([len(sent) for sent in X_train_chars])"
      ],
      "execution_count": 129,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "297.7325149966674"
            ]
          },
          "metadata": {},
          "execution_count": 129
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BzMd9IUy0a2U"
      },
      "source": [
        "char_nums = [len(sent) for sent in X_train]"
      ],
      "execution_count": 138,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "me31VMkd05Jz",
        "outputId": "981885a0-6fa9-40de-ee56-d24ca005445a"
      },
      "source": [
        "plt.hist(char_nums, bins=7)\n",
        "plt.show()"
      ],
      "execution_count": 139,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD4CAYAAADy46FuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAWqUlEQVR4nO3df6zddZ3n8edr2wF/zEqLdBimbbZ1bNxUsrNigzVuJsY6paCxbIKmxCzVYW12xV1n1kSLJkNWJYGdyTCSKA4jHYthQZZxlkZhu13EmE0W5CLKT5EroLQBe6UIu2P8Uee9f5zPhWO9/ZTec3vuFZ6P5OR+v+/P53vO+3xz73n1++PepqqQJOlw/sl8NyBJWtgMCklSl0EhSeoyKCRJXQaFJKlr8Xw3MNdOOumkWrVq1Xy3IUm/Ue68884fVdWymcZecEGxatUqJiYm5rsNSfqNkuT7hxvz1JMkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeo6YlAk2ZFkf5J7Zxj7UJJKclJbT5LLk0wmuTvJaUNztyZ5qD22DtVfn+Sets3lSdLqJybZ0+bvSbJ0bt6yJOloPJ8jis8Dmw4tJlkJbAR+MFQ+E1jTHtuAK9rcE4GLgDcApwMXDX3wXwG8b2i76dfaDtxSVWuAW9q6JGnMjvib2VX19SSrZhi6DPgwcONQbTNwdQ3+N6TbkixJcgrwZmBPVR0ASLIH2JTka8Arquq2Vr8aOBu4uT3Xm9vz7gS+BnzkqN7dUVq1/SvH8unn3KOXvG2+W5D0IjCraxRJNgP7qurbhwwtBx4bWt/bar363hnqACdX1eNt+Qng5E4/25JMJJmYmpo62rcjSeo46qBI8jLgo8CfzX07M2tHKIf9P1ur6sqqWldV65Ytm/FvWkmSZmk2RxS/D6wGvp3kUWAF8M0kvwvsA1YOzV3Rar36ihnqAD9sp61oX/fPoldJ0oiOOiiq6p6q+p2qWlVVqxicLjqtqp4AdgHntbuf1gNPt9NHu4GNSZa2i9gbgd1t7Jkk69vdTufx3DWPXcD03VFb+dVrIZKkMXk+t8deC/wf4DVJ9iY5vzP9JuBhYBL4G+D9AO0i9ieAO9rj49MXttucz7VtvsfgQjbAJcAfJXkIeGtblySN2fO56+ncI4yvGlou4ILDzNsB7JihPgGcOkP9SWDDkfqTJB1b/ma2JKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqMigkSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkrqOGBRJdiTZn+TeodqfJ/lOkruT/H2SJUNjFyaZTPJgkjOG6ptabTLJ9qH66iS3t/oXkxzX6se39ck2vmqu3rQk6fl7PkcUnwc2HVLbA5xaVf8C+C5wIUCStcAW4LVtm88kWZRkEfBp4ExgLXBumwtwKXBZVb0aeAo4v9XPB55q9cvaPEnSmB0xKKrq68CBQ2r/s6oOttXbgBVteTNwXVX9rKoeASaB09tjsqoerqqfA9cBm5MEeAtwQ9t+J3D20HPtbMs3ABvafEnSGM3FNYo/Bm5uy8uBx4bG9rba4eqvBH48FDrT9V95rjb+dJv/a5JsSzKRZGJqamrkNyRJes5IQZHkY8BB4Jq5aWd2qurKqlpXVeuWLVs2n61I0gvO4tlumOQ9wNuBDVVVrbwPWDk0bUWrcZj6k8CSJIvbUcPw/Onn2ptkMXBCmy9JGqNZHVEk2QR8GHhHVf1kaGgXsKXdsbQaWAN8A7gDWNPucDqOwQXvXS1gbgXOadtvBW4ceq6tbfkc4KtDgSRJGpMjHlEkuRZ4M3BSkr3ARQzucjoe2NOuL99WVf+uqu5Lcj1wP4NTUhdU1S/b83wA2A0sAnZU1X3tJT4CXJfkk8BdwFWtfhXwhSSTDC6mb5mD9ytJOkpHDIqqOneG8lUz1KbnXwxcPEP9JuCmGeoPM7gr6tD6T4F3Hqk/SdKx5W9mS5K6DApJUpdBIUnqMigkSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXUcMiiQ7kuxPcu9Q7cQke5I81L4ubfUkuTzJZJK7k5w2tM3WNv+hJFuH6q9Pck/b5vIk6b2GJGm8ns8RxeeBTYfUtgO3VNUa4Ja2DnAmsKY9tgFXwOBDH7gIeANwOnDR0Af/FcD7hrbbdITXkCSN0RGDoqq+Dhw4pLwZ2NmWdwJnD9WvroHbgCVJTgHOAPZU1YGqegrYA2xqY6+oqtuqqoCrD3mumV5DkjRGs71GcXJVPd6WnwBObsvLgceG5u1ttV597wz13mv8miTbkkwkmZiamprF25EkHc7IF7PbkUDNQS+zfo2qurKq1lXVumXLlh3LViTpRWe2QfHDdtqI9nV/q+8DVg7NW9FqvfqKGeq915AkjdFsg2IXMH3n0lbgxqH6ee3up/XA0+300W5gY5Kl7SL2RmB3G3smyfp2t9N5hzzXTK8hSRqjxUeakORa4M3ASUn2Mrh76RLg+iTnA98H3tWm3wScBUwCPwHeC1BVB5J8Arijzft4VU1fIH8/gzurXgrc3B50XkOSNEZHDIqqOvcwQxtmmFvABYd5nh3AjhnqE8CpM9SfnOk1JEnj5W9mS5K6DApJUpdBIUnqMigkSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXSMFRZI/TXJfknuTXJvkJUlWJ7k9yWSSLyY5rs09vq1PtvFVQ89zYas/mOSMofqmVptMsn2UXiVJszProEiyHPiPwLqqOhVYBGwBLgUuq6pXA08B57dNzgeeavXL2jySrG3bvRbYBHwmyaIki4BPA2cCa4Fz21xJ0hiNeuppMfDSJIuBlwGPA28BbmjjO4Gz2/Lmtk4b35AkrX5dVf2sqh4BJoHT22Oyqh6uqp8D17W5kqQxmnVQVNU+4C+AHzAIiKeBO4EfV9XBNm0vsLwtLwcea9sebPNfOVw/ZJvD1X9Nkm1JJpJMTE1NzfYtSZJmMMqpp6UM/oW/Gvg94OUMTh2NXVVdWVXrqmrdsmXL5qMFSXrBGuXU01uBR6pqqqp+AXwJeBOwpJ2KAlgB7GvL+4CVAG38BODJ4foh2xyuLkkao1GC4gfA+iQva9caNgD3A7cC57Q5W4Eb2/Kutk4b/2pVVatvaXdFrQbWAN8A7gDWtLuojmNwwXvXCP1KkmZh8ZGnzKyqbk9yA/BN4CBwF3Al8BXguiSfbLWr2iZXAV9IMgkcYPDBT1Xdl+R6BiFzELigqn4JkOQDwG4Gd1TtqKr7ZtuvJGl2Zh0UAFV1EXDRIeWHGdyxdOjcnwLvPMzzXAxcPEP9JuCmUXqUJI3G38yWJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUtdIQZFkSZIbknwnyQNJ3pjkxCR7kjzUvi5tc5Pk8iSTSe5OctrQ82xt8x9KsnWo/vok97RtLk+SUfqVJB29UY8oPgX8j6r658AfAA8A24FbqmoNcEtbBzgTWNMe24ArAJKcCFwEvAE4HbhoOlzanPcNbbdpxH4lSUdp1kGR5ATgD4GrAKrq51X1Y2AzsLNN2wmc3ZY3A1fXwG3AkiSnAGcAe6rqQFU9BewBNrWxV1TVbVVVwNVDzyVJGpNRjihWA1PA3ya5K8nnkrwcOLmqHm9zngBObsvLgceGtt/bar363hnqvybJtiQTSSampqZGeEuSpEONEhSLgdOAK6rqdcA/8NxpJgDakUCN8BrPS1VdWVXrqmrdsmXLjvXLSdKLyihBsRfYW1W3t/UbGATHD9tpI9rX/W18H7ByaPsVrdarr5ihLkkao1kHRVU9ATyW5DWttAG4H9gFTN+5tBW4sS3vAs5rdz+tB55up6h2AxuTLG0XsTcCu9vYM0nWt7udzht6LknSmCwecfv/AFyT5DjgYeC9DMLn+iTnA98H3tXm3gScBUwCP2lzqaoDST4B3NHmfbyqDrTl9wOfB14K3NwekqQxGikoqupbwLoZhjbMMLeACw7zPDuAHTPUJ4BTR+lRkjQafzNbktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqGjkokixKcleSL7f11UluTzKZ5ItJjmv149v6ZBtfNfQcF7b6g0nOGKpvarXJJNtH7VWSdPTm4ojig8ADQ+uXApdV1auBp4DzW/184KlWv6zNI8laYAvwWmAT8JkWPouATwNnAmuBc9tcSdIYjRQUSVYAbwM+19YDvAW4oU3ZCZzdlje3ddr4hjZ/M3BdVf2sqh4BJoHT22Oyqh6uqp8D17W5kqQxGvWI4q+ADwP/2NZfCfy4qg629b3A8ra8HHgMoI0/3eY/Wz9km8PVf02SbUkmkkxMTU2N+JYkScNmHRRJ3g7sr6o757CfWamqK6tqXVWtW7Zs2Xy3I0kvKItH2PZNwDuSnAW8BHgF8ClgSZLF7ahhBbCvzd8HrAT2JlkMnAA8OVSfNrzN4eqSpDGZ9RFFVV1YVSuqahWDi9Ffrap3A7cC57RpW4Eb2/Kutk4b/2pVVatvaXdFrQbWAN8A7gDWtLuojmuvsWu2/UqSZmeUI4rD+QhwXZJPAncBV7X6VcAXkkwCBxh88FNV9yW5HrgfOAhcUFW/BEjyAWA3sAjYUVX3HYN+f2Ot2v6V+W7heXv0krfNdwuSZmlOgqKqvgZ8rS0/zOCOpUPn/BR452G2vxi4eIb6TcBNc9GjJGl2/M1sSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqMigkSV0GhSSpa9ZBkWRlkluT3J/kviQfbPUTk+xJ8lD7urTVk+TyJJNJ7k5y2tBzbW3zH0qydaj++iT3tG0uT5JR3qwk6eiNckRxEPhQVa0F1gMXJFkLbAduqao1wC1tHeBMYE17bAOugEGwABcBbwBOBy6aDpc2531D220aoV9J0izMOiiq6vGq+mZb/r/AA8ByYDOws03bCZzdljcDV9fAbcCSJKcAZwB7qupAVT0F7AE2tbFXVNVtVVXA1UPPJUkakzm5RpFkFfA64Hbg5Kp6vA09AZzclpcDjw1ttrfVevW9M9Rnev1tSSaSTExNTY30XiRJv2rkoEjy28DfAX9SVc8Mj7UjgRr1NY6kqq6sqnVVtW7ZsmXH+uUk6UVlpKBI8lsMQuKaqvpSK/+wnTaifd3f6vuAlUObr2i1Xn3FDHVJ0hiNctdTgKuAB6rqL4eGdgHTdy5tBW4cqp/X7n5aDzzdTlHtBjYmWdouYm8EdrexZ5Ksb6913tBzSZLGZPEI274J+DfAPUm+1WofBS4Brk9yPvB94F1t7CbgLGAS+AnwXoCqOpDkE8Adbd7Hq+pAW34/8HngpcDN7SFJGqNZB0VV/W/gcL/XsGGG+QVccJjn2gHsmKE+AZw62x4lSaPzN7MlSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1LV4vhs4kiSbgE8Bi4DPVdUl89ySZmHV9q/MdwtH5dFL3jbfLUgLxoI+okiyCPg0cCawFjg3ydr57UqSXlwWdFAApwOTVfVwVf0cuA7YPM89SdKLykI/9bQceGxofS/whkMnJdkGbGur/y/Jg7N8vZOAH81y2/lgv8dILgV+g/pt7PfYeqH3+88ON7DQg+J5qaorgStHfZ4kE1W1bg5aGgv7Pbbs99iy32NrLvtd6Kee9gErh9ZXtJokaUwWelDcAaxJsjrJccAWYNc89yRJLyoL+tRTVR1M8gFgN4PbY3dU1X3H8CVHPn01ZvZ7bNnvsWW/x9ac9ZuqmqvnkiS9AC30U0+SpHlmUEiSugwKBn8mJMmDSSaTbJ/vfgCSrExya5L7k9yX5IOtfmKSPUkeal+XtnqSXN7ew91JTpunvhcluSvJl9v66iS3t76+2G5KIMnxbX2yja+ah16XJLkhyXeSPJDkjQt5/yb50/a9cG+Sa5O8ZCHt3yQ7kuxPcu9Q7aj3Z5Ktbf5DSbaOud8/b98Pdyf5+yRLhsYubP0+mOSMofpYPj9m6ndo7ENJKslJbX1u929VvagfDC6Sfw94FXAc8G1g7QLo6xTgtLb8T4HvMvgzJv8F2N7q24FL2/JZwM1AgPXA7fPU938C/ivw5bZ+PbClLX8W+Pdt+f3AZ9vyFuCL89DrTuDftuXjgCULdf8y+OXTR4CXDu3X9yyk/Qv8IXAacO9Q7aj2J3Ai8HD7urQtLx1jvxuBxW350qF+17bPhuOB1e0zY9E4Pz9m6rfVVzK44ef7wEnHYv+O9QdzIT6ANwK7h9YvBC6c775m6PNG4I+AB4FTWu0U4MG2/NfAuUPzn503xh5XALcAbwG+3L5JfzT0g/fsvm7f2G9sy4vbvIyx1xPaB28OqS/I/ctzf6XgxLa/vgycsdD2L7DqkA/eo9qfwLnAXw/Vf2Xese73kLF/DVzTln/lc2F6/47782OmfoEbgD8AHuW5oJjT/eupp5n/TMjyeeplRu20weuA24GTq+rxNvQEcHJbXgjv46+ADwP/2NZfCfy4qg7O0NOz/bbxp9v8cVkNTAF/206VfS7Jy1mg+7eq9gF/AfwAeJzB/rqThbt/px3t/lwI38fT/pjBv8phgfabZDOwr6q+fcjQnPZrUCxwSX4b+DvgT6rqmeGxGvyTYEHc35zk7cD+qrpzvnt5nhYzOIy/oqpeB/wDg1Mjz1pg+3cpgz+IuRr4PeDlwKZ5beooLaT9eSRJPgYcBK6Z714OJ8nLgI8Cf3asX8ugWMB/JiTJbzEIiWuq6kut/MMkp7TxU4D9rT7f7+NNwDuSPMrgr/y+hcH/I7IkyfQvdg739Gy/bfwE4Mkx9rsX2FtVt7f1GxgEx0Ldv28FHqmqqar6BfAlBvt8oe7faUe7P+d7P5PkPcDbgXe3cKPT13z2+/sM/uHw7fZztwL4ZpLf7fQ1q34NigX6Z0KSBLgKeKCq/nJoaBcwfafCVgbXLqbr57W7HdYDTw8d8h9zVXVhVa2oqlUM9uFXq+rdwK3AOYfpd/p9nNPmj+1fm1X1BPBYkte00gbgfhbo/mVwyml9kpe1743pfhfk/h1ytPtzN7AxydJ2FLWx1cYig/8o7cPAO6rqJ0NDu4At7W6y1cAa4BvM4+dHVd1TVb9TVavaz91eBjfAPMFc799jddHlN+nB4A6B7zK4e+Fj891P6+lfMThMvxv4VnucxeA88y3AQ8D/Ak5s88PgP3n6HnAPsG4ee38zz9319CoGP1CTwH8Djm/1l7T1yTb+qnno818CE20f/3cGd4Es2P0L/GfgO8C9wBcY3IGzYPYvcC2D6ye/aB9a589mfzK4NjDZHu8dc7+TDM7hT//MfXZo/sdavw8CZw7Vx/L5MVO/h4w/ynMXs+d0//onPCRJXZ56kiR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXf8fWfBom7qekSwAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "asrHssit1E6X",
        "outputId": "15d42754-b585-4d87-8958-076b35e6923f"
      },
      "source": [
        "#get 75% and 95 % int to use in character tokenizer output length\n",
        "txt_len_75 = int(np.percentile(char_nums, 75))\n",
        "txt_len_95 = out_len = int(np.percentile(char_nums, 95))\n",
        "txt_len_75, txt_len_95"
      ],
      "execution_count": 165,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(189, 290)"
            ]
          },
          "metadata": {},
          "execution_count": 165
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "orNB9Pkh4lV1",
        "outputId": "a4e35523-e8a8-45f0-f118-fbf07b65bdef"
      },
      "source": [
        "#max. number of tokens in vectorizer needs to be the max. number of all aplhanumeric characters plus punctuation available\n",
        "import string\n",
        "all_chars = string.ascii_lowercase + string.digits + string.punctuation\n",
        "all_chars"
      ],
      "execution_count": 144,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'abcdefghijklmnopqrstuvwxyz0123456789!\"#$%&\\'()*+,-./:;<=>?@[\\\\]^_`{|}~'"
            ]
          },
          "metadata": {},
          "execution_count": 144
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rRExT_T95eVc",
        "outputId": "3b1039b0-a1d0-4241-a6bc-b8234deaa864"
      },
      "source": [
        "#max number of tokens to use in vectorizer is the number of all_chars PLUS 2 (1 for ' ' and 1 for OOV/['UNK'])\n",
        "MAX_TOKENS =len(all_chars)\n",
        "MAX_TOKENS"
      ],
      "execution_count": 166,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "68"
            ]
          },
          "metadata": {},
          "execution_count": 166
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gFr01Jad51gg"
      },
      "source": [
        "#instantiate\n",
        "char_vectorizer = tf.keras.layers.TextVectorization(\n",
        "    max_tokens=MAX_TOKENS, #number of words in vectorizer vocab\n",
        "    standardize='lower_and_strip_punctuation',\n",
        "    split='whitespace', ngrams=None, output_mode='int',\n",
        "    output_sequence_length=out_len, #number of words each vectorized inpit nis limited to (here: 290, as this captures 95% of all sentence lengths)\n",
        "    pad_to_max_tokens=True, vocabulary=None,\n",
        "    name=\"char_vectorizer\"\n",
        ")"
      ],
      "execution_count": 167,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oqMFINaD6JNe"
      },
      "source": [
        "#adapt tp X_train\n",
        "char_vectorizer.adapt(X_train_chars)"
      ],
      "execution_count": 168,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PnSt4XlN5xSz",
        "outputId": "1ef6a758-eeda-4ca9-caee-82c7ee14bfeb"
      },
      "source": [
        "#check vocab\n",
        "char_vectorizer.get_vocabulary(), len(char_vectorizer.get_vocabulary())"
      ],
      "execution_count": 169,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(['',\n",
              "  '[UNK]',\n",
              "  'e',\n",
              "  't',\n",
              "  'i',\n",
              "  'a',\n",
              "  'n',\n",
              "  'o',\n",
              "  'r',\n",
              "  's',\n",
              "  'd',\n",
              "  'c',\n",
              "  'l',\n",
              "  'h',\n",
              "  'p',\n",
              "  'm',\n",
              "  'u',\n",
              "  'f',\n",
              "  'g',\n",
              "  'y',\n",
              "  'w',\n",
              "  'v',\n",
              "  'b',\n",
              "  'k',\n",
              "  'x',\n",
              "  'z',\n",
              "  'q',\n",
              "  'j'],\n",
              " 28)"
            ]
          },
          "metadata": {},
          "execution_count": 169
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kZlujotE7ox_"
      },
      "source": [
        "##### create embedder that works at char-level and with char-vectorizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Kt3U6lIN5mpY"
      },
      "source": [
        "#embed\n",
        "char_embedder = tf.keras.layers.Embedding(input_dim=len(char_vectorizer.get_vocabulary()), # set input shape --> = length of vocabulary\n",
        "                             output_dim=25, # set size of embedding vector --> 25 as per model in paper that I am trying to reproduce\n",
        "                             mask_zero=True,\n",
        "                             embeddings_initializer=\"uniform\", # default, intialize randomly\n",
        "                             input_length=out_len, # how long is each input \n",
        "                             name=\"char_embed\")"
      ],
      "execution_count": 170,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0J9tQhA-752X",
        "outputId": "49933b22-cd4d-49d2-d42f-c7cc6e616a89"
      },
      "source": [
        "print(f\"Indiv. character sentence: {random_text}\")\n",
        "print(f\"Sentence vectorized (with a length of {len(char_vectorizer([random_text])[0])}): \\n{char_vectorizer([random_text])}\")\n",
        "print(f\"Embedding of vectorized sentence (with a length of {len(char_embedder(char_vectorizer([random_text]))[0][0])} each): \\n{char_embedder(char_vectorizer([random_text]))}\")\n",
        "print(f\"\\nEmbedding shape: \\n{char_embedder(char_vectorizer([random_text])).shape}\")"
      ],
      "execution_count": 172,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Indiv. character sentence: occluded vision in seniors seems to prolong this period up to @ min .\n",
            "Sentence vectorized (with a length of 290): \n",
            "[[1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "  0 0]]\n",
            "Embedding of vectorized sentence (with a length of 25 each): \n",
            "[[[ 0.01040469 -0.02757432 -0.02865853 ... -0.02507263  0.04656357\n",
            "    0.00173384]\n",
            "  [ 0.01040469 -0.02757432 -0.02865853 ... -0.02507263  0.04656357\n",
            "    0.00173384]\n",
            "  [ 0.01040469 -0.02757432 -0.02865853 ... -0.02507263  0.04656357\n",
            "    0.00173384]\n",
            "  ...\n",
            "  [-0.02346617 -0.03808194 -0.0068071  ...  0.0332175   0.02434533\n",
            "   -0.02919314]\n",
            "  [-0.02346617 -0.03808194 -0.0068071  ...  0.0332175   0.02434533\n",
            "   -0.02919314]\n",
            "  [-0.02346617 -0.03808194 -0.0068071  ...  0.0332175   0.02434533\n",
            "   -0.02919314]]]\n",
            "\n",
            "Embedding shape: \n",
            "(1, 290, 25)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hfR-9w6U85_f"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}