Project Code/Portfolio/6 - Neural Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ba0abe7a",
   "metadata": {},
   "source": [
    "# Neural Networks (NN)\n",
    "\n",
    "### Pre-requisites\n",
    "\n",
    "Similarly to our previous notebooks, the following modules and/or packages are required. These have been previously explained in related notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3b66fd1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.python.eager.context import get_config\n",
    "from tensorflow import keras \n",
    "from keras.models import Sequential\n",
    "from keras.layers import LSTM, Dense, Dropout, Masking, Embedding\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from keras.models import Sequential\n",
    "from keras.layers import LSTM, Dense, Dropout, Masking, Embedding\n",
    "\n",
    "# from keras import load_model\n",
    "from keras.callbacks import EarlyStopping, ModelCheckpoint\n",
    "\n",
    "from keras.utils.vis_utils import plot_model\n",
    "import pydot\n",
    "import graphviz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "dd15bd0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import wordpunct_tokenize\n",
    "\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.stem.porter import PorterStemmer\n",
    "from nltk.stem.lancaster import LancasterStemmer\n",
    "from nltk.stem import WordNetLemmatizer, SnowballStemmer\n",
    "## nltk.download('stopwords')\n",
    "## nltk.download('punkt')\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score\n",
    "from sklearn import metrics\n",
    "\n",
    "from word2number import w2n\n",
    "import string\n",
    "import re\n",
    "\n",
    "from time import time\n",
    "import pickle\n",
    "import os\n",
    "\n",
    "import random\n",
    "from random import sample"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0bb514e",
   "metadata": {},
   "source": [
    "We change the work directory accordingly to where our data from the previous analysis was stored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "06b753b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir(\"D://Datasets//Various Datasets\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e12773dd",
   "metadata": {},
   "source": [
    "We now load the files (and close them!) using pickles. The 3 variables will contain:\n",
    "- 'data' : The lyrics, pre-processed and cleaned\n",
    "\n",
    "- 'seq' : The lyrics, sequentially transformed. Loading this is optional, since we transform them any way.\n",
    "\n",
    "- 'genre' : The target variable, i.e. the genres to be predicted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "id": "b5a38cea",
   "metadata": {},
   "outputs": [],
   "source": [
    "file_1 = open('lyrics.sav', 'rb')\n",
    "file_2 = open('sequences.sav', 'rb')\n",
    "file_3 = open('Target.sav', 'rb')\n",
    "\n",
    "data = pickle.load(file_1)\n",
    "seq = pickle.load(file_2)\n",
    "genre = pickle.load(file_3)\n",
    "\n",
    "file_1.close()\n",
    "file_2.close()\n",
    "file_3.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a11be12",
   "metadata": {},
   "source": [
    "## Recurrent Neural Net - RNN\n",
    "\n",
    "The first approach to NN implementation we take is using bidirectional, dynamic networks with the ability to back-and-forth propagate information - known as recurrent neural networks (RNN). The main advantage of such graphs is that information is perpetually improved, as well as preserved to earlier states depending on epoch, by retaining memory of the epochs in a local log. \n",
    "\n",
    "\n",
    "Unlike other NNs like Word2Vec, or techniques such as skip-grams or N-grams, RNNs do not rely on contextual analysis: which would be unsuitable for song analysis, as they do not respect context or circumstances, unlike books or novels. Therefore, a more simple approach is preferred in these situations, as long as we find a way to reliable transform data into a numerical format.\n",
    "\n",
    "First, let's look at whether the data has translated correctly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "id": "e122ec65",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "79452"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "206ebfaa",
   "metadata": {},
   "source": [
    "First, we set out threshold to the number of words we want to consider from a song. We might not be interested in songs with fewer than 40 words, for example. Let's set an initial expectation at 100:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "id": "2dadb3bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "threshold=100"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56015f14",
   "metadata": {},
   "source": [
    "We then initialize the tokenizer from keras. This will be necessary, despite having already tokenized the lyrics, so we can fit it in order to perform sequentialization of the data: i.e. getting words to numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "id": "5bdc07b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer(\n",
    "    num_words=None,\n",
    "    filters='!\"#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\\t\\n',\n",
    "    lower=True, split=' ')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8bce0eb",
   "metadata": {},
   "source": [
    "We now fit it on the entire data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "id": "584bc3d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.fit_on_texts(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "763e8ec6",
   "metadata": {},
   "source": [
    "Now we can define the sequences. This is identical to our 'seq' variable that we have imported."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "id": "17cd6307",
   "metadata": {},
   "outputs": [],
   "source": [
    "sequences = tokenizer.texts_to_sequences(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd36f75c",
   "metadata": {},
   "source": [
    "We create the dataset by binding together the sequences, lyrics and genres:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "id": "028edb5b",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.concat([pd.Series(data), genre, pd.Series(sequences)], axis=1)\n",
    "data.columns = ['Lyrics', 'Genre', 'Numeric_lyrics']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb7a0333",
   "metadata": {},
   "source": [
    "We now split the data into our usual train/test split:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "id": "38a88dba",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = data['Lyrics']\n",
    "y = data['Genre']\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 124, stratify=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b078b040",
   "metadata": {},
   "source": [
    "Lastly, we should transform the train and test accordingly instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "id": "350a50d5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[377, 13, 18, 63, 620, 213, 1486, 26, 147, 554, 399, 213, 16, 159, 12]"
      ]
     },
     "execution_count": 189,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequences_train = tokenizer.texts_to_sequences(X_train)\n",
    "sequences_test = tokenizer.texts_to_sequences(X_test)\n",
    "sequences_train[0][:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4184cdc6",
   "metadata": {},
   "source": [
    "We save the indices from the entire data (i.e. those fit on both train+test together), so we can re-trace their marking on the lyrics (using only the resulting numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "id": "e0549203",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'sorri babi take hand wing alright haha look sun search moon alright say fli yeah'"
      ]
     },
     "execution_count": 190,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idx_word = tokenizer.index_word\n",
    "\n",
    "' '.join(idx_word[w] for w in sequences_train[0][:15])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e44f43e",
   "metadata": {},
   "source": [
    "Now: One issue we encounter is that not all songs have the same length, i.e. number of words. However, what we can feasibly look at is a set threshold for the minimum number of words to compare against. Therefore, we start by looking at the number of words for each of our 2 splits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "id": "2f60103d",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_len = [len(x) for x in sequences_train]\n",
    "test_len = [len(x) for x in sequences_test]\n",
    "\n",
    "y_train = pd.DataFrame(y_train)\n",
    "y_test = pd.DataFrame(y_test)\n",
    "y_train.columns = ['Genre']\n",
    "y_test.columns = ['Genre']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "id": "7dc692c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train['ind'] = np.where(pd.Series(train_len)>=threshold, 1, 0)\n",
    "y_test['ind'] = np.where(pd.Series(test_len)>=threshold, 1, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "id": "0fe2d2c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "target_train = y_train['Genre'].loc[y_train['ind']==1].reset_index(drop=True)\n",
    "target_test = y_test['Genre'].loc[y_test['ind']==1].reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "id": "92ff61aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    2\n",
       "2    0\n",
       "3    0\n",
       "4    2\n",
       "Name: Genre, dtype: int64"
      ]
     },
     "execution_count": 194,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66719794",
   "metadata": {},
   "source": [
    "Based on the findings, we decide to set an initial threshold of 100:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "id": "b19cb43b",
   "metadata": {},
   "outputs": [],
   "source": [
    "sequences_train = [seq for seq in sequences_train if len(seq)>=threshold]\n",
    "sequences_test = [seq for seq in sequences_test if len(seq)>=threshold]\n",
    "\n",
    "sequences_train = [seq[:threshold] for seq in sequences_train]\n",
    "sequences_test = [seq[:threshold] for seq in sequences_test]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6a4867d",
   "metadata": {},
   "source": [
    "We now construct the model architecture-wise. Crucially, the first layer is an LSTM layer - which is commonly used in text predictive/classification oriented NNs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "id": "0ab1da7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "model.add(LSTM(64, activation='relu',\n",
    "               input_shape=(threshold,1), return_sequences=True))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(LSTM(64, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(1, activation='relu'))\n",
    "\n",
    "model.compile(optimizer='adam',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2ef1a4f",
   "metadata": {},
   "source": [
    "We may now take a look at the model summary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "id": "a9ae48d4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_7\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " lstm_14 (LSTM)              (None, 100, 64)           16896     \n",
      "                                                                 \n",
      " dropout_21 (Dropout)        (None, 100, 64)           0         \n",
      "                                                                 \n",
      " lstm_15 (LSTM)              (None, 64)                33024     \n",
      "                                                                 \n",
      " dropout_22 (Dropout)        (None, 64)                0         \n",
      "                                                                 \n",
      " dense_14 (Dense)            (None, 32)                2080      \n",
      "                                                                 \n",
      " dropout_23 (Dropout)        (None, 32)                0         \n",
      "                                                                 \n",
      " dense_15 (Dense)            (None, 1)                 33        \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 52,033\n",
      "Trainable params: 52,033\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be7148d9",
   "metadata": {},
   "source": [
    "We now need to fit our data. However, we would need to get it into a format accepted by the tf-keras framework. Firstly, the lists need translating to series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "id": "01db13ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = pd.Series(sequences_train)\n",
    "X_test = pd.Series(sequences_test)\n",
    "\n",
    "y_train = target_train.copy()\n",
    "y_test = target_test.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed975fd0",
   "metadata": {},
   "source": [
    "Let's check the outcome:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "id": "89bad7a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    [34, 136, 1004, 45, 136, 192, 384, 616, 112, 8...\n",
       "1    [431, 658, 264, 79, 79, 377, 56, 551, 74, 204,...\n",
       "2    [11, 48, 40, 174, 174, 11, 11, 1355, 776, 1355...\n",
       "3    [1304, 29, 23, 1579, 156, 1333, 345, 28, 86, 1...\n",
       "4    [50, 16264, 39, 1834, 97, 89, 2, 20, 86, 11, 2...\n",
       "dtype: object"
      ]
     },
     "execution_count": 199,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19759073",
   "metadata": {},
   "source": [
    "We perform a quick match between our splits, to ensure we retained the correct dimensions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "id": "e11fe8bc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sanity test outcome for training: True\n",
      "Sanity test outcome for testing: True\n"
     ]
    }
   ],
   "source": [
    "print(f'Sanity test outcome for training: {y_train.shape == X_train.shape}')\n",
    "print(f'Sanity test outcome for testing: {y_test.shape == X_test.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddee390b",
   "metadata": {},
   "source": [
    "Then, we transform the lists output above into a readable format for the NN - by translating them into columns for a dataframe. We add them together to construct the necessary 'final' testing and training sets respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "id": "98c0f068",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(threshold):\n",
    "    globals()['X_good_{}'.format(i)] = [int(X_train[j][i]) for j in range(len(X_train))]\n",
    "    globals()['X_bad_{}'.format(i)] = [int(X_test[j][i]) for j in range(len(X_test))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "id": "280e8fb2",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_final_train = pd.concat([pd.Series(globals()['X_good_{}'.format(i)]) for i in range(threshold)], axis=1)\n",
    "X_final_test = pd.concat([pd.Series(globals()['X_bad_{}'.format(i)]) for i in range(threshold)], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6dec5837",
   "metadata": {},
   "source": [
    "We are now prepared fit the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "id": "4cd6858d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1056/1056 [==============================] - 103s 95ms/step - loss: 5.1467 - accuracy: 0.3091 - val_loss: 2.5913 - val_accuracy: 0.2631\n",
      "Epoch 2/5\n",
      "1056/1056 [==============================] - 106s 100ms/step - loss: 3.2820 - accuracy: 0.2820 - val_loss: 1.6029 - val_accuracy: 0.2578\n",
      "Epoch 3/5\n",
      "1056/1056 [==============================] - 114s 108ms/step - loss: 2.7585 - accuracy: 0.2727 - val_loss: 1.5956 - val_accuracy: 0.2579\n",
      "Epoch 4/5\n",
      "1056/1056 [==============================] - 112s 106ms/step - loss: 2.6386 - accuracy: 0.2722 - val_loss: 1.5956 - val_accuracy: 0.2579\n",
      "Epoch 5/5\n",
      "1056/1056 [==============================] - 113s 107ms/step - loss: 2.4313 - accuracy: 0.2694 - val_loss: 1.6029 - val_accuracy: 0.2578\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_final_train,y_train,epochs=5, validation_data=(X_final_test,y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf7d1778",
   "metadata": {},
   "source": [
    "We can notice that, from one epoch to the other, the loss decreases - indicating a trend of progressive learning. However, the primary evaluation accuracies decrease as well, leading to very similar overall accuracies in the end. There are many steps that could be tried given these tendencies: first of all, providing more data. Clearly, the information gain of the RNN is weak to the point that architectural choice wouldn't notably infleunce the outcome.\n",
    "\n",
    "Nevertheless, results from this model seem to be underwhelming, as a result of deep recursive learning of noisy data. Therefore, it is preferable we take an alternate - and more basic - approach to the problem."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a749740b",
   "metadata": {},
   "source": [
    "## Simple MLP (Multi Layer Perceptron)\n",
    "\n",
    "Perhaps somewhat counterintuitively, previous approaches seem to indicate that we might have better success by using less powerful tools, but more cleverly. The most basic type of neural networks is an SLP (Single Layer Perceptron), which is made of only one input, one ouput and a single hidden layer in between.\n",
    "\n",
    "An MLP simply adds more hidden layers to the process, with the option to drawback on a simple SLP by implicit architecture choice. This offers a wider range of hyper-parameter tuning potential, as well as more flexibility. Let's start by setting a threshold for the # of words selected from sequences, just like we did for RNN:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "id": "e0e18a01",
   "metadata": {},
   "outputs": [],
   "source": [
    "threshold=200"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a287f4e7",
   "metadata": {},
   "source": [
    "We will then define our feature+target based on the threshold. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "id": "46c60de9",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Get the initial data as uploaded\n",
    "X = seq\n",
    "y = genre\n",
    "\n",
    "## Select only data with more than threshold words per song\n",
    "X_good = X[X.str.len()>threshold]\n",
    "idx = X_good.index\n",
    "y = y[y.index.isin(idx)].copy() ## Set the target to match our selection\n",
    "\n",
    "## Reset the index to avoid extra, unwanted column\n",
    "X_good = X_good.reset_index(drop=True)\n",
    "y = y.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d767357",
   "metadata": {},
   "source": [
    "We now need to bind the words into singular values, bu column. To do that, we transform them into columns 1 by 1 rather than an array, to create M features for the task where M = threshold chosen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "id": "6ac0f8ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(threshold):\n",
    "    globals()['X_good_{}'.format(i)] = [int(X_good[j][i]) for j in range(len(X_good))]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ba2f0c1",
   "metadata": {},
   "source": [
    "We then transform them all into series, so they can later be combined into a single dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "id": "2f43c031",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(threshold):\n",
    "    globals()['X_good_{}'.format(i)] = pd.Series(globals()['X_good_{}'.format(i)])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d189d68a",
   "metadata": {},
   "source": [
    "Output a sample of our methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "id": "981813d5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    39991\n",
       "1      966\n",
       "2        5\n",
       "3       49\n",
       "4      372\n",
       "dtype: int64"
      ]
     },
     "execution_count": 210,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_good_0.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a10928b8",
   "metadata": {},
   "source": [
    "We can now concatenate them into the final data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "id": "f86edd83",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_final = pd.concat([globals()['X_good_{}'.format(i)] for i in range(threshold)], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "id": "4de8a225",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11088, 200)"
      ]
     },
     "execution_count": 212,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_final.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40f524c0",
   "metadata": {},
   "source": [
    "Lastly, check that the shape of the target matches our expectations before performing our split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "id": "2e2cdfb9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11088,)"
      ]
     },
     "execution_count": 213,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a62beece",
   "metadata": {},
   "source": [
    "Perform the train/test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "id": "64d68308",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X_final, y, test_size = 0.2, random_state = 124, stratify=y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64fb52fa",
   "metadata": {},
   "source": [
    "Finally, we define our MLP model with the chosen architecture, and fit our data on it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "id": "05bcf31d",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = MLPClassifier(random_state=1, max_iter=300, activation='relu', learning_rate='adaptive',\n",
    "                   hidden_layer_sizes=(15,10,6,3,3,6,10,15)).fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95eebbb4",
   "metadata": {},
   "source": [
    "We output the accuracy score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "id": "ee522603",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7091974752028855"
      ]
     },
     "execution_count": 216,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07a9ef09",
   "metadata": {},
   "source": [
    "This falls short off our expectations for using it as a general model. At this level of performance, we would not be justified in considering a NN approach (either RNN or simple MLP) as opposed to the more basic vectorized logistic regression. However, this can be explained by considering what happens at each input of new word thresholds from songs. It is likely that performance will keep increasing with data at an exponential rate once the NN achieves its learning balance.\n",
    "\n",
    "To test this hypothesis, we will perform a self-built grid search which re-initializes the entire procedure with a free choice of hyperparameters. We will pay particular consideration to how different activation functions perform on the given task.\n",
    "\n",
    "Let's deploy some empty lists/arrays where we will store our findings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "id": "462da73a",
   "metadata": {},
   "outputs": [],
   "source": [
    "relu=[]\n",
    "tan=[]\n",
    "logi=[]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37f6a64c",
   "metadata": {},
   "source": [
    "We now define the gridsearch accordingly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "id": "8b3895e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def GridSearchNN(threshold, activ='relu', max_iter=200, hid=(15,10,6,3,3,6,10,15)):\n",
    "    ''' We initialize the data just like before, the only change being the threshold that can now be pre-set '''\n",
    "    \n",
    "    ## Get and combine the data\n",
    "    X = seq\n",
    "    y = genre\n",
    "    X_good = X[X.str.len()>threshold]\n",
    "    idx = X_good.index\n",
    "    y = y[y.index.isin(idx)].copy()\n",
    "\n",
    "    X_good = X_good.reset_index(drop=True)\n",
    "    y = y.reset_index(drop=True)\n",
    "    \n",
    "    ## Perform the datapoints binding \n",
    "    for i in range(threshold):\n",
    "        globals()['X_good_{}'.format(i)] = [X_good[j][i] for j in range(len(X_good))]\n",
    "    for i in range(threshold):\n",
    "        globals()['X_good_{}'.format(i)] = pd.Series(globals()['X_good_{}'.format(i)])\n",
    "    \n",
    "    ## Combine the resulting columns in a format accepted by the model\n",
    "    X_final = pd.concat([globals()['X_good_{}'.format(i)] for i in range(threshold)], axis=1)\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X_final, y, test_size = 0.2, random_state = 124, stratify=y)\n",
    "    \n",
    "    ## Perform prediction\n",
    "    clf = MLPClassifier(random_state=1, max_iter=max_iter, activation=activ, hidden_layer_sizes=hid).fit(X_train, y_train)\n",
    "    score = round(clf.score(X_test, y_test)*100, 4)\n",
    "    print(f\"For threshold: {threshold} ~~~~~ Activation: {activ} ~~~~~~~\\\n",
    "    Iterations: {max_iter} ~~~~~~~ Score: {score}\")\n",
    "    \n",
    "    ## Append the resulting accuracy scores for each \n",
    "    ## activation function to their respective list\n",
    "    if(activ=='relu'):\n",
    "        relu.append(score)\n",
    "    elif(activ=='logistic'):\n",
    "        logi.append(score)\n",
    "    else:\n",
    "        tan.append(score)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0d49002",
   "metadata": {},
   "source": [
    "All that's left now is to apply the GridSearch on several parameters of our choice. Most importantly, we will test the NN across various thresholds, i.e. words per song. The expected result is that it will initially have _some_ performance, that will keep dropping as it gets to see more words which add unprecedented variety - and this trend might continue until the learning rate will be able to catch up to it; i.e. when words start repeating sufficiently for the NN to pick it up as signal.\n",
    "\n",
    "We will of course also monitor performance across our 3 main activation functions. Off-record, a large number of other properties were tested as well, including maximum iterations permitted, learning rate, architecture as well as randomness. Neither of these exhibited particular propensities one way or the other for us to delve deeper into. Therefore, we will run a slightly less complicated grid-search this time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "id": "4c62020c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "For threshold: 1 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 58.0621\n",
      "For threshold: 1 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 58.3517\n",
      "For threshold: 1 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 58.0621\n",
      "For threshold: 5 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 58.0334\n",
      "For threshold: 5 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 58.0334\n",
      "For threshold: 5 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 58.0334\n",
      "For threshold: 10 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 58.0167\n",
      "For threshold: 10 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 58.023\n",
      "For threshold: 10 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 58.0167\n",
      "For threshold: 20 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 57.8834\n",
      "For threshold: 20 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 57.8834\n",
      "For threshold: 20 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 57.8834\n",
      "For threshold: 30 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 57.5953\n",
      "For threshold: 30 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 57.6534\n",
      "For threshold: 30 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 57.5953\n",
      "For threshold: 40 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 56.9336\n",
      "For threshold: 40 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 56.9932\n",
      "For threshold: 40 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 56.9336\n",
      "For threshold: 50 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 55.7714\n",
      "For threshold: 50 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 55.7783\n",
      "For threshold: 50 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 55.7714\n",
      "For threshold: 60 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 54.0133\n",
      "For threshold: 60 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 54.0133\n",
      "For threshold: 60 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 54.0133\n",
      "For threshold: 70 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 51.6851\n",
      "For threshold: 70 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 51.7179\n",
      "For threshold: 70 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 51.6851\n",
      "For threshold: 80 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 48.9644\n",
      "For threshold: 80 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 48.9644\n",
      "For threshold: 80 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 48.9644\n",
      "For threshold: 90 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 45.7942\n",
      "For threshold: 90 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 47.6936\n",
      "For threshold: 90 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 45.7942\n",
      "For threshold: 100 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 41.9401\n",
      "For threshold: 100 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 41.9642\n",
      "For threshold: 100 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 42.0844\n",
      "For threshold: 110 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 37.8699\n",
      "For threshold: 110 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 46.3289\n",
      "For threshold: 110 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 37.9676\n",
      "For threshold: 120 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 34.1527\n",
      "For threshold: 120 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 34.1527\n",
      "For threshold: 120 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 37.7177\n",
      "For threshold: 130 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 38.8678\n",
      "For threshold: 130 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 49.734\n",
      "For threshold: 130 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 41.3944\n",
      "For threshold: 140 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 43.9019\n",
      "For threshold: 140 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 50.5745\n",
      "For threshold: 140 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 43.7251\n",
      "For threshold: 150 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 48.9216\n",
      "For threshold: 150 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 48.9216\n",
      "For threshold: 150 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 48.744\n",
      "For threshold: 170 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 58.7639\n",
      "For threshold: 170 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 60.5952\n",
      "For threshold: 170 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 58.7639\n",
      "For threshold: 180 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 63.5424\n",
      "For threshold: 180 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 64.5756\n",
      "For threshold: 180 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 63.5424\n",
      "For threshold: 190 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 67.5133\n",
      "For threshold: 190 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 67.3494\n",
      "For threshold: 190 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 67.5133\n",
      "For threshold: 200 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 70.9197\n",
      "For threshold: 200 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 70.9197\n",
      "For threshold: 200 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 70.9197\n",
      "For threshold: 210 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 74.4522\n",
      "For threshold: 210 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 74.4522\n",
      "For threshold: 210 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 74.3526\n",
      "For threshold: 220 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 77.3224\n",
      "For threshold: 220 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 77.3224\n",
      "For threshold: 220 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 77.3224\n",
      "For threshold: 230 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 79.8913\n",
      "For threshold: 230 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 79.8913\n",
      "For threshold: 230 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 79.8913\n",
      "For threshold: 240 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 81.7757\n",
      "For threshold: 240 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 81.7757\n",
      "For threshold: 240 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 81.7757\n",
      "For threshold: 250 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 83.5424\n",
      "For threshold: 250 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 81.9926\n",
      "For threshold: 250 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 83.0996\n",
      "For threshold: 260 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 85.1608\n",
      "For threshold: 260 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 85.1608\n",
      "For threshold: 260 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 85.1608\n",
      "For threshold: 270 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 86.484\n",
      "For threshold: 270 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 86.484\n",
      "For threshold: 270 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 86.484\n",
      "For threshold: 280 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 87.6656\n",
      "For threshold: 280 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 87.6656\n",
      "For threshold: 280 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 87.6656\n",
      "For threshold: 290 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 88.4835\n",
      "For threshold: 290 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 88.1414\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "For threshold: 290 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 88.4835\n",
      "For threshold: 300 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 89.3671\n",
      "For threshold: 300 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 89.3671\n",
      "For threshold: 300 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 89.3671\n",
      "For threshold: 310 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 90.1146\n",
      "For threshold: 310 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 90.1146\n",
      "For threshold: 310 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 90.1146\n",
      "For threshold: 320 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 90.0958\n",
      "For threshold: 320 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 90.0958\n",
      "For threshold: 320 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 90.0958\n",
      "For threshold: 330 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 90.4676\n",
      "For threshold: 330 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 90.4676\n",
      "For threshold: 330 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 90.4676\n",
      "For threshold: 340 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 90.9836\n",
      "For threshold: 340 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 90.9836\n",
      "For threshold: 340 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 90.9836\n",
      "For threshold: 350 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 91.8415\n",
      "For threshold: 350 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 91.8415\n",
      "For threshold: 350 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 91.8415\n",
      "For threshold: 360 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 91.9355\n",
      "For threshold: 360 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 91.129\n",
      "For threshold: 360 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 91.9355\n",
      "For threshold: 370 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 92.1212\n",
      "For threshold: 370 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 91.8182\n",
      "For threshold: 370 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 90.9091\n",
      "For threshold: 380 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 92.8826\n",
      "For threshold: 380 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 92.8826\n",
      "For threshold: 380 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 92.8826\n",
      "For threshold: 390 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 92.2449\n",
      "For threshold: 390 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 92.2449\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\AnacondaIsHere\\lib\\site-packages\\sklearn\\neural_network\\_multilayer_perceptron.py:614: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "For threshold: 390 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 92.6531\n",
      "For threshold: 400 ~~~~~ Activation: logistic ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 92.5581\n",
      "For threshold: 400 ~~~~~ Activation: relu ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 92.5581\n",
      "For threshold: 400 ~~~~~ Activation: tanh ~~~~~~~    Iterations: 200 ~~~~~~~ Score: 92.5581\n"
     ]
    }
   ],
   "source": [
    "thresh = [1,5,10,20,30,40,50,60,70,80,90,100,110,120,130,140,150,170,180,190,200,210,220,230,240,250,260,270,280,290,300,\n",
    "         310,320,330,340,350,360,370,380,390,400] ## Set threshold markers\n",
    "activ = ['logistic', 'relu', 'tanh'] ## Set activation function markers\n",
    "for item in thresh:\n",
    "    for act in activ:\n",
    "        GridSearchNN(item, activ=act, max_iter=200) ## Perform the search iteratively and output findings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8357990",
   "metadata": {},
   "source": [
    "We will perform a quick test that our results have been transcribed accordingly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "id": "3bf0acb3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 220,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Should evaluate to True\n",
    "len(relu)==len(tan)==len(logi)==len(thresh)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eba206cf",
   "metadata": {},
   "source": [
    "We'll bind our results into a single dataframe so it can be plotted accordingly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "id": "fc827e8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat([pd.Series(thresh), pd.Series(logi), pd.Series(tan), pd.Series(relu)], axis=1)\n",
    "df.columns = ['X', 'Logit', 'Tan', 'Relu']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "id": "c6f84e19",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiYAAAGpCAYAAAC5wP3WAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABUMElEQVR4nO3deZRc533e+e/v1t5dvW9oAA009oUgCJDgvoiLVmuhFEmO5JEjexxrJuOZ2MnJxMqcnHEmJ5kjJzMZT5xMMrIniRJLtmXZimTRoiWRoihK3ABuAEHse6PR+1p71X3njyqAAIilAXT1ra5+PufgVNWte289ty9Z/ev3vvd9zTmHiIiISC3wgg4gIiIicp4KExEREakZKkxERESkZqgwERERkZqhwkRERERqRjjoAHPR2dnp+vv7g44hIiIi82DPnj2jzrmuK723KAqT/v5+du/eHXQMERERmQdmdvJq7+lSjoiIiNQMFSYiIiJSM1SYiIiISM1QYSIiIiI1Q4WJiIiI1AwVJiIiIlIzVJiIiIhIzVBhIiIiIjVDhYmIiIjUDBUmIiIiUjNUmIiIiEjNUGEiIiIiNUOFiYiIiNQMFSYiIiJSM8JBBxAREZEbV/SLFHJpcq/sIdG3Gnq7icYbMbOgo90SFSYiIiKLSKFUYCI9wnBqkPZSA/lEielT75A99AovtJ8mFo7TFGuhKdZKU6yFZKyFZU19tCY6go4+JypMREREalypWMAfGyfS00OhmOXw2D4yhRSJiUbCvuE3xEh3hFgdT5ArZpjOTXBu5gyhbJ6i57hn3QfZueJBAA4Mv8FPjj1FxIsS9aI0+lFiRCklY8RCMR5Z+1F854iF44S80IIfqwoTERGRGlUsFRk7e4T82bPE9h0n+Zkn+cvjf0amkCYSinLv6k9ip4dID5+l5Z476Yk68qUchVKefClH5Ow4iTPjhN4coRAdIdLVRb6UwzmfxskcD51qIup7DG9sZqDFI+yFmcxOYIBn7YS8xIIfswoTERGRGuGco+RKhL3yr2fPPPIjw/h+ERfyOPLsnzPSPUhLvJ0nb/siyVgz9PSRvGj7fClHtpghl8+QKU1S7Gkj8vYpcA7nHMua+vjE1i/SmDPyJ74PQLM1EutYje8cES9COBTB84K5P0aFiYiIyAJwzqfkfHxXwvcrj5XXJefj++df+3Q19uB5ITzPo2PZWmaf+j5FzzEYmqI51sYnt/1quSi5jJkRC8eJheMU8yEmRlMUcmnCsTjhri7MjJCFIWQ0tLeT9zwsFKJhtkRvywZC4UjgnWdVmIiIiMyBc47i0BDpI4coUGJ8ZSP5mEe+lClfOvHzhCyMA3qb+ljbsRmAifQYA1PH6WjswbMrt0I4wGUy+Nks0fYOin6RaKV/R8PylRzclOQVDhBvaOFT236VpljLdfOGm5vp+vDHKE1PU5qdvVBwtDV0XsjR8Uu/hCUSgRcjF1NhIiIich2+7zN2fD+5o0cptjYCxrnCCIVCkQMjr+OcD8Dqto00RJqYyo5e2PbY+H4mM+OMpofoTvbSnVxB2AvjWQjPPDzzSP34J5RGRgkXirR87ONEk7FLPn/z/U9y9ojj4TUfoTneekPZQ83NhJrfbV25uDjyGhpu/IdRZSpMRERErqJYKnJ0fD+D06fAd6zpbMEKRaLhGPFYiKjz2dF7P5FQlKgfovnMLKUVHbS19l/YR3dyBcfHD3Bu5jQAzdE27ln9GBu7tl8oEkoNzWTTAwCkX3qZyMc+RskVCVkYMyMRaeCjW35pwY8/CCpMRERELlKammJ67xucXm6M+pM45wAwDC8apWnKp2HjGnpXrLhku8w77zC7/wDsP0N0fQoe7wGgr3UtyxMrGNj3c9KHDjIYn+VH+b/gtYEXeGTtR1nZsobY+vXkjh4ltmYNsY0bKZYKPHXgG7TE23l03cewq1wCqkcqTERERCqm9+xmeugUpfYm3GwRl3B45tGTXM6GztuJhKNX3TZ76NCF5+Hu7kveK549S2LPYRJ4tJa6OBX1GE8P4/slACLLltHxhS9g4TBFv8BT73yDM1PHGE8Pc3ffo1fs6FqvVJiIiMiS5JzDzCiWinheua9HtKuLUn4c5xmtowWat21gdefmOd0623jnnWQPHSJ/6hTxdesueS/a14dFo7h8nlA6x+fW/jKn3RB9reX1zIzXh15mWVMfu8/8hNOTR0lEGvnktl9ZUkUJqDAREZElxPk+mf37SZ0+TmFslKMPLCNfyrGx83b62tYR61tFZO9rhMIR2m67k0jXijnfsRLt6yPa14crFrHwpb9eLRym4Y47IBQitnYtoWSS9XRdeH80NcTPTvyA8v05EA838MnbfoX2hktbXpYCFSYiIlL3iqUCh8feZiI1zMpXjpEoekSA/MwUNCQYTQ/R17YOM6P3gx/DIpGb/qzLi5LzGnbuvOo2yWgTO1c8wFuDLxPxonxy26/Q0dhz0xkWMxUmIiJSV3KnTjF78hi5qTHid92J39xA0S8yOjtIvpRnuN1n9XD50syaXDudm+6mOd52YftbKUpuVjzSwIP9H+KulY9gQCy88EPB14qqFiZm9pvArwMG/IFz7vfMrB34U6AfOAH8onNuopo5RESkvvi5HMWxMYhESDcao+khprMTpPOzNE2VaInFoLuFYmqcWDKKZ0Z7Qze+K7Fs+zoa0xBduZLOlpaaGlwsvoQLkvOqVpiY2TbKRck9QB542syeqix7xjn3FTP7MvBl4LerlUNERGqPn8uROXqYzOQYkTu3YxgNkQY8C+Hncsy+vgcAL56g4Y47MDOcc0xkRvGPnsJ7+U0ActvXcrAjdcm+o9EwlnN46Rxeepi2NduIeFG6k8sX/DjlxlWzxWQL8JJzLg1gZj8BPgU8CTxaWedrwHOoMBERWTLGjx8gvW8vflsThXiIE6efB2BD5+3EwjH8fJ5cOAdA6cQxvp/5LgCdDcvoSq4giWNtZV+RiTTWaYS9CA2RRprjbbTTRDw/QXRlF5GeHiy08Jdm5OZVszDZB/xzM+sAMsAvALuBHufcIIBzbtDMll6XYxGRJWgiM8a+wVcoFvP097RhAKVS+VKK48Lw7A7DSj5Ulpfnn/EBK6+bbADPI9zWRijZxqNrH3zv7bxd/Qt/gDIvqlaYOOfeMbPfBX4IzAJvAsW5bm9mXwK+BLBq1aqqZBQRkepwpRL5kyeZObif9sfejxePEw8lKPklfIN8MUPTWJ7m/g2sW7/9km39cJZscgacw27r4+9s/uJ7999fwkKhhTocWUB2fqjdqn+Q2f8OnAF+E3i00lrSCzznnNt0rW137drldu/evRAxRUTkFuWKGYaf/h4u2YA5R2v7Cpru2IlzjlOTR+lsXEaDF4dQqKY6nsrCMbM9zrldV3qv2nfldDvnhs1sFfA3gPuBNcAXga9UHr9TzQwiIrIwhmfO8ubgSxwZ28dqEtwW34FzjvThgzTdsRMzY3Xb+qBjSo2r9jgmf17pY1IAfsM5N2FmXwG+aWa/BpwCPlvlDCIiUiW+75OaGOb1qdfIFjIcHn0Lh6O0soPI8TGSK9fS+KH7g44pi0hVCxPn3MNXWDYGPFHNzxURkeoq+SWy0+NMHj9IsVQg1h6l6BXYufxBblu2i5ZEO+4Op0s1csM08quIiMxZtpDm4MhbjKWGWZtuhUIeK5boGwrTuuv9NEQaL6yrokRuhgoTERG5rqnsBIeG32I6N3lh2Wyj0XF4hNDEDA1bbyMR0qilcutUmIiIyFUNTZ/h6Pg7ZAtpzt/DGfEirGxdS3/bBjJTbxB9pI9It4akkvmhwkRERK7IOcfpqeN46Ryr0gnGk46+5VvobX53bKnGu+4KMKHUIxUmIiICwHR2gsOjb9MSb2N9522YGStLbZSODRMeHaajtZ3WTX1Bx5Q6p8JERGQJc84xMHSQM4dexUJhZrsbGEsNsb7zNgC6ezcw9oOfA1CanqY0NUW4tTXAxFLvVJiIiCxBuWKGwyNvc3j4Le5/pcA6Z5QSMY53t9PXswHnyrf6erEY4a4uwu3tNN57L148HnR0qXMqTERElghXKuE8I5Wf4czkcYZmBwhHokw2ZOlIhQllcuz01hNvXXvJdq1PPoldPkmeSJWoMBERqXPOObLHjzH59hvY3XdgjQlaEu2Mpc/R17qW3i15SgNnifT2Em5vf8/2KkpkIakwERGpc6OvvUh2coxSTwt27izJDZtpj3ex/PzdNV3Anbq7RmqDymARkTqVL+bYc+YF9idHKTTHsZJP5NQQzdkQkVA06HgiV6QWExGROuOc4/j4AU5MHMY5Bx6kQwV6sw20ffDj6sAqNU2FiYhInSjNzjL70kucWeY4F0sD4JnHxq7bWbGhP9hwInOkwkREpA7kBweZ+v73oViktdDJyLow7U093NZzFyFPX/WyeKiPiYjIIjcwdYK3cgfxYjEAImfG2OmvZXvvvSpKZNHRf7EiIotU0S+w+/TzpPOzFP0iK3ZsILb/JMkHHyS6fHnQ8URuigoTEZFFpjQ1xdjZI/wo+yLj6WFWtW5gectqOpftILxll8YdkUVNhYmIyCLhnCP19j4mjx/AxqfI943T0tLB3X3vu2TGX5HFTIWJiMgiUSzmGJkewHraCDUmeN9omBWPfJ5oOBZ0NJF5o/Y+EZEa55wjlZ9lPDtOZN1arFAkNJtl5eMfU1EidUctJiIiNcoVi0zmJzg8+jarWtdjZjS3dBFdnSB233IsFAo6osi8U2EiIlKDMseOMrbvNU6uj5ON+IykzrKxazuxcBxWtgUdT6RqdClHRKTGpE4eZ/To2xSXtdGdjtEQSrC2Y2u5KBGpcypMRERqSKGU59jMEYotDVjJp2Fgkrtbd9EYTQYdTWRB6FKOiEiNyBdz7D33KpOxNKNeiS0DUbre/1FCjY1BRxNZMCpMRERqhJlRLBVIF2bYufxBlm1dE3QkkQWnwkREJEDOOaBclERCUe5c+SAT6VG6mzSkvCxNKkxERALiO5+p7ARjqXOsKLUTb+8kEomqKJElTYWJiEhAin6BoZkzDIwfJf7qBIlQnOQDDxLr7w86mkhgdFeOiEhATk0c5pVTz9JwYIBYpoQ/m2L2Zz/DFYtBRxMJjFpMREQWUKaQwrMQ52ZO8YNDf47vSrT03IVNT+GyWZL334+F9dUsS5f+6xcRWQDOOWZz06QLs6Tys/zocLko2bH8Abb0fwi3M0f2yBGia3QnjixtKkxERKrMOcdUdoJcMUOumOWV08+SL2XZ2nMXD/Z/CDPD4nEatm0LOqpI4FSYiIhUke98JjPjFEo5zDy6kr20JTppS3Tw6LqPY2ZBRxSpKSpMRESqpOSXmMyMUfQLgNGW6CQSivCBjZ8ms+9tXDoDGtVV5BK6K0dEpAqKfpGJzCiFUp7BmdP85Oh3gfJgasWBs6RffJGJb36T9N69FwZZExG1mIiIzLtCKc9kZox0IcWxsXc4MPw6JVfizOQx+ts2MvuznwHgCgWKw8O6nCNyERUmIiLzKF/MMZEZZWR2kOMTBzk9eZimWCvv3/ApeptXA5B8+GFmX3gBP5Uief/9AScWqS0qTERE5km+mOPczAADU8c4N3uGs1PH2bbsbh7o/yCRUPTCetHly2n79Kcpjo/jNTQEmFik9qgwERGZJ+FQhGxxloHp48zmpvj4bb/Mqtb1V1zXQiEiXV0LnFCk9qkwERG5Bc45CqU80XAMzzzWddxGrphlY9ftxMIJcqdOURgYwE+liG/aRLSvL+jIIjVNhYmIyE3yfZ8jh18kN3SWnjGjadNWElu3cnvvPRfWKQwMkNm7F4BwZ6cKE5Hr0O3CIiI3oeSXePbId5jOTdAca8XNzFAcH3/Pel4y+e42s7MLGVFkUVKLiYjIDSr5RZ4++E2Ojx9gQ7GXlWca8bIF/CsUHpHeXhrvvhuvsZFwZ2cAaUUWFxUmIiI3oFDM88yRb3N8/ACxcIId6z9IS28JL5kk1NT0nvUjnZ1EVJCIzJkKExGROXDOkXr9dQ5GhwjHYrQnuvnAxk/TleyFnqDTidQPFSYiItfhnGN296tMzgzTkUxQtBJ3nC9KRGReqfOriMh1FHNZpoozlJJxrFBkw2CEzsZlQccSqUsqTEREriGdn+Wls89RWNtLCI/mtNH26BOa30akSnQpR0TkKlL5GV459Ry+8zk9e4J7tzxAtKEJC4WCjiZSt6raYmJmf8/M3jazfWb2x2YWN7N2M/uhmR2uPLZVM4OIyI1yvs/U2FlerRQlnnnsWPEAsaZWFSUiVVa1wsTMVgB/F9jlnNsGhIDPAV8GnnHObQCeqbwWEakJzvcZ+/EPGDvwJl4RPPPY1fcIzfHWoKOJLAnV7mMSBhJmFgYagLPAk8DXKu9/DfhklTOIiMzZ6E+eIR0DLxJh5ajHrq57aYq1BB1LZMmoWmHinBsA/g/gFDAITDnnfgD0OOcGK+sMAt1X2t7MvmRmu81s98jISLViiohc4JxjZk0HpWgIL52jjSTJZs0ALLKQqnkpp41y68gaYDnQaGZfmOv2zrmvOud2Oed2dWlqcBFZAGZG/+odWGc7zYk2Wh98RHffiCywat6V837guHNuBMDM/gJ4ABgys17n3KCZ9QLDVcwgIjInuWKWWDiOZx79a++CtUEnElmaqtnH5BRwn5k1WPlPjieAd4DvAl+srPNF4DtVzCAick2uWOT42AGePvCnHB7ZF3QckSWvai0mzrmXzexbwGtAEXgd+CqQBL5pZr9GuXj5bLUyiIhci3OO8Wd/RKbNp6WljZncZNCRRJa8qg6w5pz7HeB3Lluco9x6IiISqPT+t/FPnKJjKErDsmZWPrgz6EgiS56GpBeRJev04NsAeJk8LZFWwg2NAScSERUmIrIkHRs7wCutZ/npqjH87nZaH3ok6EgigubKEZElKFfMcnLiMCta1tLe20n38gd0W7BIjVCLiYgsKc45UvkZ1rRvoinWwh2996koEakhajERkSWjMDzM5MhpSn09NEab2dW3Hs/095lILdH/kSKyJPjZLCMvPkcqNUHh5EmaXFxFiUgN0v+VIrIkjO/dQ6G7FQDv+CARp68/kVqk/zNFpO4VSnn2tU0wliwSmkzRfvudhJqbg44lIlegPiYiUvdeO/MCJVdiKuGz6a77SHT0BB1JRK5ChYmI1LVDI3uZzc8AsKV7J43NKkpEapkKExGpW9NDpzgzcxyAnuQKljX3BZxIRK5HfUxEpC7ljh9n9qWX6E7HiVqErT13Bh1JROZALSYiUnf8TIaZn/6UcDZLR6FI74o+vPX6O0xkMdD/qSJSd5zziSxbhgHRrE/bnfcEHUlE5kgtJiJSV5xzvDT8M/x1BXaueohoshkvFgs6lojMkQoTEakrx8cPki1kKLki+VVdJBu6go4kIjdAl3JEpG7ki1leOP5XTGXH6E4up11FiciioxYTEakL2SNHeGdwN9M2STzSyPZe9SsRWYxUmIjIoleanWXitZdZMZXm4WQnKz70C3gWCjqWiNwEFSYisuhNvbGH/OpuLFek88wQHU3Lgo4kIjdJfUxEZFFzzlG4fS3pxjCWztDx6AewSCToWCJyk9RiIiKLWjo/C55HZ/8WGntDxDvU4VVkMVNhIiKLVq6QYSY3hZnRHG8llowHHUlEbpEu5YjIolTK5Xhn+A2Oju2nWCoQC6soEakHajERkUWnODnJ0M+fxe9LMBOZxSzoRCIyX1SYiMii4nyfiRd/SqGjmfYMJM85lm/uDzqWiMwTXcoRkUXFz2aZ6YiDQXhkkr47Hg46kojMIxUmIrKojPtTnGzLMBKexTraSXT3Bh1JROaRLuWIyKLhO5/j4wfxcaRawqxY93jQkURknqnFREQWjZJfJBltpuSK3LVCl3BE6pFaTERkUSilUkQaG9nSs5ONXdsIh6JBRxKRKlCLiYjUvMLICOPf+AYzzz+Pn8moKBGpYypMRKSmOeeYeu0Vcn2dpI8eZvbnPw86kohUkS7liEhNc4UC080eXqgJSiU67ror6EgiUkVqMRGRmpZ2GX4ceosjNkioq5NwW1vQkUSkitRiIiI17cWTPyJTSFHoiNK7+dGg44hIlanFRERq1uD0KQ6OvIlnIR7o/0DQcURkAajFRERqUmFykmPj79DftpmOhi5a4u1BRxKRBaAWExGpOX46zcgLz9A1HaLJj7NzxUNBRxKRBaLCRERqzszreyh0NBHNw+aTRiwcDzqSiCwQFSYiUlOcc+SSUVwoRGg6Tdv2XZhZ0LFEZIGoj4mI1JTZ3DSn2nN0R1fQcnaWyMqVQUcSkQWkFhMRqSn7h19jKjvBidQxmneqtURkqVFhIiI1Y2jmLKncNA7H1h6N8CqyFOlSjojUhMLEBIcn3sIByVgznY09QUcSkQCoxUREAudKJUae/xE94x6NOY/be+4OOpKIBESFiYgEbnbfXortSSK+x8pjGRIWCzqSiAREhYmIBG7KS1OMh/FyBVr7N+FFo0FHEpGAqDARkcAt33YfDWs3EEo00HjbtqDjiEiA1PlVRALnmceyjn7o6A86iogErGotJma2yczeuOjftJn9lpm1m9kPzexw5bGtWhlEpLaNpYY5NLIX51zQUUSkRlStMHHOHXTO7XDO7QDuAtLAt4EvA8845zYAz1Rei8gSU5qd5cDgHo6PH+D1gZ8HHUdEasRC9TF5AjjqnDsJPAl8rbL8a8AnFyiDiNSQ4R//NcvPlujMNbChfWvQcUSkRixUH5PPAX9ced7jnBsEcM4Nmln3lTYwsy8BXwJYtWrVgoQUkYWRO3Oa8OAYjYOQiEdIrHTQEHQqEakFVW8xMbMo8Angz25kO+fcV51zu5xzu7q6uqoTTkQCcWLsILORIgYkVvUTbm8POpKI1IiFuJTzEeA159xQ5fWQmfUCVB6HFyCDiNSIdCHFwfAgP9o4Tnb7GpK7NMqriLxrIQqTz/PuZRyA7wJfrDz/IvCdBcggIjXi4NAbNEab2bhsJyvvfT+hZDLoSCJSQ6rax8TMGoAPAP/dRYu/AnzTzH4NOAV8tpoZRKR2FP0iy5pXEQnH6WrowcyCjiQiNeaGChMziwNR59z0XNZ3zqWBjsuWjVG+S0dElhA/n2c6NYIX9ljVupbmuIYwEpH3mnNhYmZ/G/hlwDOznzrn/pfqxRKRejPy5stkyBNpaaWhb0vQcUSkRl21j4mZffyyRe93zr3POfcw8NHqxhKRepKfmSKbnQHnsP1HKQ6cDTqSiNSoa3V+vcPMvmNmd1Rev2VmXzezPwLeXoBsIlInjo7sJ9MYwssXifkhYuvWBR1JRGrUVS/lOOf+mZktA/5ppYPa/wokgQbn3FsLlE9EFrlMIcVgaRjXBGvinTSv36FOryJyVdfrY5ICfgvYAHwVeBX4l1XOJCJ1ZN/gbhwQ9sL0b74fz1uomTBEZDG6Vh+TfwY8RXmivcecc58A3gSeMrNfXqB8IrKIjaeGmc5NArCpa7uKEhG5rmt9S3zMOfcI8ADwtwCcc98FPgRo/GgRuSbnHEdO7QGgIZJkWXNfwIlEZDG41qWcfWb2X4AE8JPzC51zReD/rnYwEVnccsePsfx0ntmuBnpXbAw6jogsEtfq/PoFM7sdKDjnDixgJhFZ5Jzvk37lVWLT00TPhAjlmqBbLSYicn3X7PzqnNu7UEFEpH4UcxnC3d2UpqfxvBANO3YEHUlEFomqzpUjIkuP7/v8+NTThPrC3L3lQ8QyJbx4POhYIrJIqDARkXl1YuIgIS+MYSS6e4mEokFHEpFFZE6FiZmtAFZfvL5z7vlqhRKRxck5x56BFyj5Rbb23KmiRERu2HULEzP7XeBvAvuBUmWxA1SYiMgF+TNnGD3+NsN2mni0kS3dO4OOJCKL0FxaTD4JbHLO5aqcRUQWKT+bZfq55whn0jwR7yZ37xa1lojITZnLMIzHgEi1g4jI4pV5+22KniOzdTXxZBublt8VdCQRWaTm0mKSBt4ws2eAC60mzrm/W7VUIrKoNOzcyUB0hkimSGbjMuLJ1qAjicgiNZfC5LuVfyIiV1TwC7T0byCfTbO2dUXQcURkEbtuYeKc+9pCBBGRxSuVnwGgrbmHeKQh4DQisphdtTAxs286537RzPZSvgvnEs657VVNJiI1rTQ9Tai5mXPTZ5jNT9EUa6Mhmgw6logsctdqMfnNyuPHFiKIiCwehdFRJv/rfyW+cSNHVqTI+Xm6k730NC0POpqILHLXmsRvsPJ4cuHiiEitc8UiM88+C75P6uRxWpt7GGqEtR1bgo4mInVgLrcLi4hc4HyfcHs7APneVsbjeZpjrTRGmwJOJiL1QHPliMgN8aJRmp54gkxvC+cYphAqsrNnR9CxRKROXLfFxMw+ZmZqWRGRC8yMIw0TzESLNMWaaYq1BB1JROrEXAqOzwGHzexfmJkuIosIo7PnyBYzAGzu3hFsGBGpK9ctTJxzXwB2AkeB/2hmL5rZl8xMF5RFloji5CT5gQGcK48ckIg00J1cTmuig+Z4W8DpRKSezKmPiXNu2sz+HEgAvwV8CvifzexfO+d+v4r5RKQGpHfvJnfsGOGeHpIPPEBjVxfblu3CvXeIIxGRWzKXPiYfN7NvA89SnszvHufcR4A7gH9Q5XwiErDixAS5Y8fKz4eGLhQjZoan7mciMs/m8q3yWeD/cs5td879S+fcMIBzLg38t1VNJyKBs2iU+JYt4HmwegVPD/81bwz8POhYIlKn5nIp53eAwfMvzCwB9DjnTjjnnqlaMhGpCaHGRpoefpiGnTt5ffBFGmkmXUgFHUtE6tRcWkz+DPAvel2qLBORJSQdKfHWxBvM5CbZ1Hl70HFEpE7NpTAJO+fy519UnkerF0lEatHewZdJF2ZIRBJ0JJcFHUdE6tRcCpMRM/vE+Rdm9iQwWr1IIlILCsPDOL/cWFryi7wz/DoA23vvCzKWiNS5ufQx+e+Br5vZvwEMOA38raqmEpFAlVIpJr/7XbzGRhp27uR0e56uxuUU/QLLmvqCjicidey6hYlz7ihwn5klAXPOzVQ/logEKfPmm+D7+DMzZA8cYHxHE43RZnqSKzCzoOOJSB2b0wBrZvZR4DYgfv5LyTn3T6uYS0QC5DU2YrEYLpcjeucO+tvCpPIzrGheHXQ0Ealz1y1MzOzfAw3AY8AfAp8BXqlyLhEJUMMddxDfsoXcsWMUuluxQoquxl7ikYago4lInZtL59cHnHN/C5hwzv1vwP2ALjKL1DkvGiW8fi0zuSmgPD+OiEi1zaUwyVYe02a2HCgAa6oXSURqxeGxtzk4/CaD06eIhDRKgIhU31z6mPylmbUC/xJ4DXDAH1QzlIgsPFco4JzDi75bgIzMDuBwxMKJAJOJyFJyzcLEzDzgGefcJPDnZvY9IO6cm1qIcCKycNJvvUVm3z4S27aR2LaN4dwwRb+EAes6tgQdT0SWiGsWJs4538z+T8r9SnDO5YDcQgQTkYXj5/Nk9u3D5XKk9+wh1NzMiehpAJKxFqLhWMAJRWSpmEsfkx+Y2adNgxeI1C1/ZgaLlYsPr7kZVvWSzpeHLFrbvjnIaCKyxMylj8nfBxqBopllKY/+6pxzzVVNJiILJtzRQfsv/iK5o0exSITD4wdwQNiL0Kl5cURkAc1l5NemhQgiIsEyzyO+YQMAhYEBDKO3WSMDiMjCmssAa49cablz7vn5jyMitWB7731MZEZpjbcHHUVElpi5XMr5ny96HgfuAfYAj1clkYgsGFcqYaHQe5aHvBCdjT0BJBKRpW4ul3I+fvFrM+sD/kXVEonIgnClEuN/9mdEly+nYccOQs3NZPIphlNnWdW6jvJoASIiC+tmvnnOANvmO4iILKzc8eP409NkDxxg8i//Euf7HB7dxztDb/DT408HHU9Elqi59DH5fcqjvUK5kNkBvDmXnVdGjP1DyoWMA/5b4CDwp0A/cAL4RefcxA2lFpFblj99+sLz+ObNYMbQ7ACRUJQVzf3BBRORJW0uLSa7Kfcp2QO8CPy2c+4Lc9z//w087ZzbDNwBvAN8mfJoshuAZyqvRWSBNb3vfbR89KPEt2whtn49A1PHOTjyBiOps/S3bww6nogsUXPp/PotIOucKwGYWcjMGpxz6WttZGbNwCPArwA45/JA3syeBB6trPY14Dngt28mvIjcPPM8oitWEF2xAoB9B/4aKA+oFvLm8tUgIjL/5tJi8gxw8QxeCeBHc9huLTAC/Ecze93M/tDMGoEe59wgQOWx+0obm9mXzGy3me0eGRmZw8eJyM2azU1zavIwhsfWnruCjiMiS9hcCpO4c272/IvK84Y5bBcG7gT+nXNuJ5DiBi7bOOe+6pzb5Zzb1dXVNdfNROQmHBl7m3Udt7G15y6SMQ3qLCLBmUthkjKzO8+/MLO7gMwctjsDnHHOvVx5/S3KhcqQmfVW9tULDN9YZBG5FcXRUQqjozhX7tPunCMaihMJRdW3REQCN5cLyb8F/JmZna287gX+5vU2cs6dM7PTZrbJOXcQeALYX/n3ReArlcfv3ExwEbk5qddeI3/iBF5TE02PPILr6aCjoZuuxmV0NFzxyqqIyIKZywBrr5rZZmAT5Qn8DjjnCnPc//8EfN3MosAx4Fcpt9J808x+DTgFfPamkovIDXOFwoXbhP2ZGbzGRmYLKQAaokk8772jwIqILKS5jGPyG8DXnXP7Kq/bzOzzzrn/53rbOufeAHZd4a0nbjSoiNw6P58ntmYN+ZMn8ZqaSMV9BiZP0JroJBGeS9cxEZHqmsulnF93zv3b8y+ccxNm9uvAdQsTEaktocZGmh9/HFcq4c/O8uboXiYyY0xlx1nRsjroeCIic+r86pmZnX9hZiEgWr1IIlJtFgpBUyOT2XEAVrauDTiRiEjZXAqTv6bcJ+QJM3sc+GOgbibSKJw7hysWg44hsuBOThzBOYeZx8rmNUHHEREB5nYp57eBLwF/h3Ln1x8Af1DNUAvFz2aZfOopLBwmvmkTjXfffcUp4EXq0eD0KQDaE514nmYSFpHaMJe7cnzg31f+YWYPAb8P/EZ1o1XfzMG3KTbGMOeYGTlLKnWWkBfBMw/P8widOIc/MgZmxNb0E1+56pLtSzMz4PtYLFb+9+4VL5Ga4goFJr/3PaKrVxNbu5ZMAnKl8nBE6zq3BpxORORdc5oQw8x2AJ+nPH7JceAvqphpwaS9PKn1PYR9Y6Qhx/TQa5e8v45uiJfvjE6lTzB0+HVmctOkClPEwnFuOxElGWnCy+Zpv+1O4hs24Ps+pyaP0BxrJT44CZkcoWSSyLJleInEFVKIVF/u1CmKIyMUR0bIHT3K6QdX4hxEQzGaYi1BxxMRueCqhYmZbQQ+R7kgGQP+FDDn3GMLlK3qwmv6GRx+g0TaJxOHkIUqo2GWR8T0Sg5XLIEZJfNxQK6UZjxdHqx2Y3EVxZ4mQrNZLBYDoOjnOTr2DgDLZ+I0pnzsXIGJ9BFCHe30Nq+iJd6OZx6ZvXtxxSLh9naiK1diYU2cJtWRP378wvPY2rU0x5Nki1m6GnsDTCUi8l7X+k14APgp8HHn3BEAM/t7C5JqgXQle+lKXv2LOX/2LKXwNDjH8p4VbG6IkStmyJWy5IoZ3CtvwfAkXraAF48DUPSLREIRin6RsG/4sQjEIkyFMuRnzhALxSmU8pgZmdQ5QjNpvOOHaEhvoXXrHQt16LLEJB95hGh/P7ljx4itW8ea1lZWta7DM/WpEpHaYufny3jPG2afotxi8gDlu3D+BPhD59yCd9/ftWuX271790J/7C2bfuM1MjNj5HNZxja2k3EZVrduJBKKUHI+2QPvgANvYoa/SrxBZ+tKNnTeztqOLTTFWigMDRHu6sLUMVFEROqIme1xzl1pANart5g4574NfNvMGoFPAn8P6DGzfwd82zn3g2qErSfNO+7k/DytfZe9VyoVGY2dIZOaYNrLQCTMuZnTpPKzjKaGaPGSrH7mKBaLEVu7luRDD6lzrdyybCHNy6d+zPqOraxo1S3CIlJ75nJXTgr4OuU5b9opz23zZcq3DctNCoXC9Gy/98LrjaUcJ8YPMTB1HN85mqd8AFwuR2ZmgtPDr9PftpGGaDKoyFIHjo4doOgXOTDylgoTEalJN9Tb0jk3Dvy/lX8yj6KhGBu7bmdj1+1kCmkKBw5TaBzGT6UYXR5jcPo0g9OniYZibOi8je54DxYKadwVuabi5CQWDhNKlgvakxMHCXkR+ts2BJxMROTKdBtIDUpEGkjcfgdu23aKQ0NMuWEimTMUSwXypRxvD73G9IhH20CaloceIdqrOyvkylKvvkr++HHCPT0Ud2zk2Pg7RENxHl/3iaCjiYhckQqTGmZmRJYtYz3LWM92ZnPTvDP0OoXZGaLhOOmVMQo/f46uRz9IpKMj6LhSY1yhQP5UeXTX4tAQx6bKyzd23U4krOmuRKQ26XaPRSQZa+buVe9jvfViqQwu5JHfspqZhE+hlA86ntQYP5MhsmwZmOF1tHGKYUIWZmvPXUFHExG5KrWYLELdW+6isGIDI2++THhjH4VSnvH0CGOpIVa3b6AhktQdPEKouZnWj34UP5Ph2Lm9tBejtCe6NKiaiNQ0FSaLVKS5meUPfwDf+aTyMwzNDDA4c5pzM6fZ8E6WthXradyxQ51jBYvHSbT30JlyxMKa00lEapsu5Sxynnk0xVpoi3cQCUWITqSJn5sis2cPI9/6Jq5UCjqiBKzg5wl7YZY3r2ZTl0YXFpHapsKkTnQke3io/8OsGXy3EexQ4zivDvyUkq/iZCnLFNIAxCMJtZaISM1TYVJHPM9j+cc/Tfy+e8glo0yvaWMmN8XewZeDjiYLyBUKTD39NJkDB8inZzkw/AZjqWGioVjQ0URErkt9TOqMeR5N23eQvP0O3NRxjo8fZHP3zgvvO+f0V3Ody506Rf78v9XdzK41pjLjbOnRZRwRqX0qTOqUmdHXupYVzf14lUkAZ996k+zwOToee786xdax/PHjADhgsicBZOls7MFMDaQiUvv0TVXnzhclmSOHmBg8TjruGPnBU/h5jXtSrxrvu4/G++6DlcuYiBfwXYkNnbcHHUtEZE5UmCwRuZFhXCyCHw0z0RlhaHYg6EhSJaFkkobt2xna2UsuVCISitAUbwk6lojInKgwWSJa7nuQ1uYeStkMgy153hl7i+PjB4OOJVVS8ktMZsYAWNmyNuA0IiJzpz4mS4SZ0XznLhK5LEODPyVTSHN87AD5YpZN3eoUWW8GZ06RL+Uo+nnWtG8KOo6IyJypxWSJicTi3LfqcZrjrTjgzNQJjh95hfzZs0FHk1tUmp3FFQoAtCe6WNm6hr6WdYQ8/f0hIouHvrGWIM8LcXff+3jz7EsU0ilscpaRgy/Rkb2D+Np1QceTm5R66SVyJ08S7eujcdcuNnVtxzkXdCwRkRuiFpMlbHvvvfQdy0LJp9DVzNjJgxTzuaBjyU1wxSK5U6egVCJ/4sSF5RqzRkQWGxUmS5iZ0Xn/IzSMzGIYw/1JXjv386BjyU0ozc4SamrCmTGztpOfTbxIOj8bdCwRkRumwmSJCzU10fnEh4guW8Z4cYKZ3DT7BncHHUtuULi1lfbPfhb3sUc5tyJCppAiHIoEHUtE5IapMBG8RILeVbexvGU1g9Mn+Mmx73Fm8ljQseQmHM2e4Ez+LI2xZs2NIyKLkgoTuWBL9w7WdmzB4fOjt7/J2AvP4UqamXixKPkl3hl5jeHZATZ03hZ0HBGRm6LCRC5x3+r3szmxkYdOdZDOTjPx0+d0Z8cicWLiIJlCivaGbnqSK4OOIyJyU1SYyCU887jH20psWS+l5gYys5MUh4eDjiVX4YpFZn7yE7InTzA4dYq2RBdbu+/U3TgismipMJH3aLpjJ3GLERqbpnXzNiI9PUFHkqvInzpF9uBBJl7+GV2THu0N3Wzs2h50LBGRm6YB1uQ9zIzOhx+nNDlJuKMj6DhyDblj5U7KxY5mYs1t9Dd10xBNBpxKROTmqTCRK7JQ6EJRki9mefX086xsXcPqtg0BJ5OLJXbswLU0kQ5nibd2sKqjP+hIIiK3RIWJXNfBkb1kixmOju6n4dgIndvvxiIaI6MWRDo7sWSIWCFNPJzAM12dFZHFTd9icl239dxJwmIsm42TSU8x8eyPcL4fdCwBnHMcHH6LY2PvaKRXEakLKkzkujwvxO1uNcnpEn4sQipcJH34UNCxBJjJTTGbn2YqO06j+paISB1QYSJz0rRhCw3RJFYsMdoVZl/8bNCRljQ/m8X5Pqcnj+I7n0goTEuiPehYIiK3TIWJzFnHvQ8R6elmPJFnNj/DweE3g460ZM2+8AKjf/onMDJOuASrWtUpWUTqgzq/ypyZ57F83U5mhj3ShTQt8Q4KpQIRTRa3oFyxSO7UKfKdTTRmjEE/zbqOLUHHEhGZFypM5IZt6r6D6ewkmUKK6ewEzfkIkba2oGMtGaWpKYhEKLU3UfB8GprbCXn6X1lE6oMu5chNScaaCeGRPnuaoZ8/S+7kyaAjLRnhjg6Sv/g3GOyEoegsG7ruCDqSiMi8UWEiN8Uzj/CBE/jjExS7Whjf8yJ+Oh10rCUjFk2wtf9+epdvpiu5LOg4IiLzRoWJ3LTm23fijU5hJZ/xWI50IRV0pCXDM4+meAvrOrcGHUVEZF6pMJGb5sXjdN1xN+eyA/y45SjPnvk+zrmgY9U05xy5U6eY/vGPSe3ZgyuVbngf2UKaYqlQhXQiIsFTYSK3JL5yFbc98GlikQSnJo/w1uDLQUeqaS6XY/qHPyR3+DDpPXuYfeGFuW9bLDLzyiscPrWH7+3/BgeG36heUBGRgFS1MDGzE2a218zeMLPdlWXtZvZDMztcedTtHItcMtbMY+s+QXuim5HZcwzPaPC18y5vQfLiceIbymOOWCxGw44dc95X/vRpUofeoWksx8ZMJ03RlvmMKiJSExaixeQx59wO59yuyusvA8845zYAz1ReyyK3vvM21nZswcwYO76f1GuvBR0pMM458gMDTD39NJl9+97zfmLbNjCj6bHHCLXMvbjIHTtGKJ0jceAMqalhlrf0z2NqEZHaEMSlnCeBr1Wefw34ZAAZpAp29tzHxoEInS8dI717N/mzS7PlJHf0KFNPPUX+1Cky+/a9Z8LDcHs7Xb/+68RWrXrPtn4ud9X9xjdtYrQ7TDGfJb5iJWY279lFRIJW7cLEAT8wsz1m9qXKsh7n3CBA5bH7Shua2ZfMbLeZ7R4ZGalyTJkP0VgDTal3L12kl2irSay/H4vFAPBnZigOD89pu1IqxcS3vkXq1Vev2Ik439XEs12n+O6mc6xZd++8ZhYRqRXVHi7yQefcWTPrBn5oZgfmuqFz7qvAVwF27dqlWz0WATOj6dFHmfjWtyj293JgRYGduWmSseagoy0oC4dJ3H47fjpNYts2wq2t193GFQpMP/00fipF+vXX8TMZmh555N33nePE+CFWt20k5IWW3M9URJaOqraYOOfOVh6HgW8D9wBDZtYLUHmc25+TsiiEGhtp/+xnOdxXIu+KvD7wM/zLLmUsBY133knTQw/NqSiBcuFhDQ3lF2bE1q275P2CnydbzBANRdmkkV5FpI5VrTAxs0Yzazr/HPggsA/4LvDFympfBL5TrQwSDK+hgTuW34+ZkS/leXtoT9CRqqo4Pk72yBEKQ0P42exN7cOLRmn50IeIb91K8qGHiK5YceE9VyySKaTpa13Hbct20d+umYRFpH5V81JOD/DtSge9MPAN59zTZvYq8E0z+zXgFPDZKmaQgDTHW1nVuo6TE0cYmT3L0OBhunvWYV79DZ2TO3mS9KuvApDYvp3kfffd1H7M82h66KH3LJ/+6fNMtYXxWlro6l6LZ6FbyisiUsuqVpg4544B72lzds6NAU9U63OldqzvvI3R2XNEZnJkBw8z+eYRWt73KF4iEXS0eeXPzFx4Hkom53Xf5daSFH4+gR0/Rah5JTTO60eIiNSU+vvzVWrKbZkeuiYNF4+Syc2SPXIk6EjzLtzZSbS/n3BHB6E59imZs1CIdGe5EvFn04Ta2+d3/yIiNabad+XIEpfctJXp0QF8ILe6i8iWTUFHmneJrVtJbK3OZHr5bIqC5xP2S7jV3Rq7RETqnlpMpKrM81jx8IdoXL6K+MpVzOZnLhmjwxWLmvjvGsbyY5xpTLG3cZDV2x4MOo6ISNWpMJEF0b58HaF4gnwpS6aQurB89uc/Z+qv/orS9HSA6WqTc46h2fLoue2N3URC0YATiYhUnwoTWRAhL0RjtJGR1CCvnH6ObCFDfmCA7IEDFAYGGP/Wt1ScXCadn2U6O062mGZz946g44iILAj1MZEFEw83Mpo6R8kv8ebgS9w+2QFm4BzRlSvxmpqCjnjDCsPDFM6exWtqItzRMecB1ebi7PRJin4R53y6Gnvnbb8iIrVMLSayYEJeiM1ddzCaGmTP6edJr+um9cknifT20vTQQ4uyY2dhYIDUK68w88wzZA/MecaF6/KdT76Uw4C+1nWL8mcjInIzVJjIglrW3Edvcx8+JZ47+peEuzpp/fjH8c4Px77IlKo0holnHlt6dnLPqsfY0rNz3vYrIlLrVJjIgrt31RM0RJIMz55l/9Dr73nfOUdhaIj82bMBpLsx0b4+Etu2EV29et7HGPHMIxlrJhqKzet+RURqmfqYyIKLheM8tOYjnBg/xNDMGfrbN9IYLfcvKQwPM/2jH+HPzhLu6iL6qU8FnPbaYmvWEFuzZl73WSwVODlxmP72jYQ8/S8qIkuLWkwkEOs7biMeSRDywhRKuQvLQy0t+Ok0AMWRkSV5p86pyaMcHX+H773z9aCjiIgsOP05JoHwPI+7+x4lForjXTSxnxeLEV25ksLQELH+/uACBsQ5x/DsACELs7xpddBxREQWnAoTCUwi8m6HV9/3AYfnhUg+8ghePF6XMxFfT7aY4bWBFwh7ET6/838IOo6IyIJTYSKBm0iPsO/cbprjbdyx/D5Ci+QOnfzgIPljx/Camoj09BDp6bnlfR4aeRPflVjWtIamWOuthxQRWWRUmEjgJrPj5Et5xlJDTGUnaIm3BR1pTornzpF5+20AErfffsuFSbFU4NDIXgC29tx5y/lERBajpddWLjVnTfsm4uEEDtg7+Mp73q/VSf4uGcNkHkatHZw5TXtDDz3JPta0b77l/YmILEYqTKQm3N57DwbkilmOjZVHUC1OTJDavZuJb36T4sTELX+Gc47S7Cz5M2fwc7nrb3AdsQ0baLz3XuJbtxLu7r6lfZ0f6TUeTrC8ZZVuExaRJUvfflITmuOtdDYuYyR1jhMTh1jZsobsnj3kjh0DIHf0KOFdu27pM6a+/30KZ86UP+/DHya2atUt7S/a20u0d37msMkVMySjzWzu3klLonVe9ikishipxURqxm3L7iLkhXDOse/cq8TWrr3wXv7UqZveb2l2Fuf7l1xuKc1DC8x8yhTKY7c0RBsJe5GA04iIBEctJlIzQl6YDZ23cWhkH+FQlHDfSmLr1xPr7yd6k60bzjmmf/CDdy/dhEKEOzvxEol5TH5riqUCQzMDJGMtxMLxoOOIiARKhYnUlBUta4iE4jjnk/GzND/++C3tr3DuHMXR0fILz6PjC1/Ai9fWL//R9BBnp08yk5ukt+mX8EJqyBSRpUuFidSc1kQHE+kR0oVZGqKNeBa66X35qRQWi+FyOeKbNs1bUZI/fZrM/v2EmpqIrFhBbPXNjdLqnOPcTLnfS0uinXBIl3FEZGlTYSI1JxqKEgnFGJk9y5nJ49y58sGb3le8cikoe+gQkeXL5y1jcXSU/MmT5RdmN12YZItppjLj5IoZ7lh+37zlExFZrFSYSG1yPgNTJ/BdibH0MB0N3ZSmpynNzhK9wQLDwmESW7fOa7zS7OyF57cyhsnA1EmKfoGiX2B5s+bGERFRYSI1qb2xm8ZYE6cnj7L3yHPccTRCcWQEL5mk/fOfx8xuar9+Pk9pYoLixAThtrabHq01cfvtRJYvx5+ZIXKTtwyX/BIT6RHAsbJlzU0fk4hIPVFhIjVrS/dOXjvzU4ZLp7htag0A/uwsxZERItcZ0Mw5d8Vf9Nm33yb16qsAJLZtu+nCJNzaSri19aa2PW8yM8ZMborZ3BT3r37/Le1LRKReqPu/1KzmeCvbeu/G92C41QfPm9Ntw65UYuLP/5zUq69SSqcveS90UTExH6PJ3oqSX6Q53kpTvJWGaDLQLCIitUItJlLTdq18hFMTRzizvpG2B++iq2vDdbfJHTtGaXyc9Pg42cOHL7n0E2prI9TeXm7xWLas2vGvqbtpOZ3JZVCbUwGJiARChYnUtESkkdt67mIsPcLRmSMsn2Nhcl588+ZLLumEW1tp/8xnqpL1ZnjmgbqWiIhcoMJEat6Gru2Exw6wufuOOa3f/IEPkD9xgsz+/fN+Nw5A7sQJ0q+9htfURGzVKuKbNt3Q9r7z2Te4m8Zokv72TYS8mx+nRUSk3qgwkZrXGE2yrXfuE/iZ5xFbu/aSuXbmU2lysjya7OgoocbGG95+MjPGWHqIY+P76WjsoTXRUYWUIiKLkwoTWVRmpobJHD9K9PQoDXfddcNjmsyH0szMhefeDY5hUvQLvHjyh3gWoiGSVFEiInIZFSayaAxMnWTs+H46BtLY4Bi5o0dvqjDxs1nyAwOUJiexSISG7dtvaPvGXbuIb9hAaXaWcGfnnLdzzvHjI9/l+PgBktEWPrP91280uohI3dPtwrJodDZ0k0kYxc5m/EiI3IkTOFe+pcXPZpl54QWKk5PX3U9pZoaZZ54hvWcP2XfeueEcXiJBZNky4uvX39BYJq8NvMDBkTcJexE+uuWXSMaab/izRUTqnVpMZNGIRRK0tC5jNj3Eua1J7rztIxfuuMm88w7Z/fvJ7t9PYvt2kvddfd6Zi4uJ0vQ0rlTCQtXtgHp87AAvnXyGzsZedq18hK7kzY0WKyJS71SYyKKytmMLr6SHyBZjHJg5wO3Je3ClEtm3376wTri9/Zr7sEiE2IYNeIlEecA1V/2BRFoSHfQ297Gx8w7aEl1V/zwRkcVKhYksKtFwjJ6mlZyaOMKJ8UNs7t5B2IvQ9PjjZPbupTg6Smzduuvup/mxx27q86821P31tDd08fCaj1JyJRKRhpv6bBGRpUCFiSw6/W0bGJk9Czj2Dr7CnSsfIrp8OdHly/Hz+apelskdO8bsz35GqKmJ2Nq1NNxx9bFVSn6R4+MHWd95GyW/iO9KGEZchYmIyFWp86ssOpFQlGVNfTREmuiIdVE4d+7Ce140WtXP9mdmcNksxZER/FTqqus553j2yHd5+uCf8sqpH5MplOfsiYfj5dFeRUTkivQNKYvSisRKVqYb4fhpJp75Ia5UWpDPLc3OXnh+rTFMynfgvEHYi9DftulCYZKI3PiAbCIiS4ku5ciiFE00YgPDuJYG8p1N5E6fJt7fP+ftne+T2beP0uQkpZkZWn7hF+bUdyT5wAM07NhBaWaG0FUKk+NjB3jx5I8A+MDGT9OSaGcyM0bICxMJVbdFR0RksVNhIouSmdHUtowJf5psdxOHpg+wnf65b+95pF9/HZfLAeDPzl610Lh8u1AySSiZvOL7o6lz/ODQtwDHfavfz7qOrUxmxgGIhxtuquOsiMhSoks5smg13XY7oVwRVypxtjRCvpS7oe1DbW0XnpfmMDDb9aTzszz1ztcp+Hk2dd3BXSsexnc+uWIWgEQkccufISJS71SYyKLlJRIsu/9xzrYVOFccZv+5PTe0fWLzZhrvu4+Wj3yEcHf3Lecp+gXCXoSeppU8tv4TmBnZQhpwRENxQp4aKEVErkfflLKoRcJRVrWtY++5l3n97M+5vfeeORcA8Y0bb+izXKkEzmHhK++/Od7GZ7Z/iZJfJOxFcM5d1OlVtwiLiMyFChNZ9PrbNtLe0M14epiDI2+yteeuqnxO7sQJZp55Bq+hgdiGDSTvvRcA53yscgtwLBy/sH7RL1D0C5h5lywXEZGr06UcWfTMPO5a8TDLm/s5Mrof3/lV+Rx/Zqb8mE7DRbcnP3vkO/zg0LeYyU1dnoxYOEEiok6vIiJzpcJE6sLqtg10NS4nHm5gaOZMVT7Dz+ehUmCcH8NkLD3MgeE3ODK6D9+/dCyVSChCa6KdZFSzCIuIzJUu5UhdiIUTrG5bT3OsldaGzjlvl967l8KZMxQnJ2l+7DEiy5Zddd3kPffQuGsXfiqFRSIAvHzyGRyObT1305K48uSBai0REZk7FSZSF8yMVW3rb3i74sgI+dOny88nJq5ZmEBlHJNKa8m5mdMcG3+HsBdhV9/7LllvNjdNJBQlGoqpMBERuQFVv5RjZiEze93Mvld53W5mPzSzw5XHtuvtQ+RG+H6Jgcnjc1o31Np64fmNjmXyUmV01+2999EYfXdwtqJfJJWfYTIzjsPd0D5FRJa6hWgx+U3gHeD8hfYvA884575iZl+uvP7tBcghS0CxVOTFkz8klZ/B80L0Nq+65vqxNWsIt7YSam0l1NIy5885PXmUM1PHiYXi3LnyoUve88wjGWvGOacJ+0REblBVvzXNbCXwUeAPL1r8JPC1yvOvAZ+sZgZZWkJeiKZYK5FQjLeHrj/gWritjdjatYTb27FQ6Krr+fk8penpC5MFDs8OYBh3rnyIePjSEV0982iMNpGMqdOriMiNqnaLye8B/xC4eBKSHufcIIBzbtDMrjjkppl9CfgSwKpV1/6rV+Q8M2N12wYmMqNkChlGU0N0Nvbc8n7zp08z88wzQHlgtrsefZQ17ZtpirXe8r5FRORdVWsxMbOPAcPOuRsbJ7zCOfdV59wu59yurq6ueU4n9aw10UFLvJ1YOMG+c6/Oyz7Pj2ECYNHyDMHtDd3vmS14JjvFbG6mamOpiIjUu2peynkQ+ISZnQD+BHjczP4IGDKzXoDK43AVM8gSZGb0ta7FMFK5GaYqs/tej/N9XLF45Tc9D6+xEQdMhjI4995Orb5fIl1IkcpPX/F9ERG5vqoVJs65f+ScW+mc6wc+BzzrnPsC8F3gi5XVvgh8p1oZZOnqaOyhOd5KPNLA3uu0mmSPHGH8W99i9D/+R9Kvv37FdRq2b6fl87/I9++Y4S/zz3Ny4vB71skUM4AjFo4T8q7eX0VERK4uiFsGvgJ8wMwOAx+ovBaZV555rGhZQ9iL0JrouPbKvk9pfBxKJYrXuGV437lXmSlO0ZrsZvVlY6aUJ+xLARDXhH0iIjdtQQZYc849BzxXeT4GPLEQnytLW0/Tigu36xZK+ff0Bznv4rFMXDZ7xXXyxRy7Tz8PwH2rn7gwad95Bb9AyS/imUcspAn7RERulkZ+lbrlmUdDtIl0foZUfuaqLSfh9nZan3ySUGsrXix2xXXeOPtzssU0y5r66G/b9J73sxe1lmikVxGRm6fRn6SuNUQa8Z3PmanjV71Dx8JhIj09Vy1K0jNjnD70Kg35EPf3PfGewsN3PplCBoBEpHF+D0BEZIlRi4nUtZAXIlNIMTQzQLaQYnP3DsJe5Ib2cWT/T3nkWCvQSjR/GD609pL3s4U04IiGYoQ9/S8lInIr1GIidW9V6zqioRgz+SkODL1xw9uvifZdeH5+Ar/znHOkK5dx1FoiInLrVJhI3QuHIvQ0rWQ0NchrAy/gu9IV13PO4Wcy77kzJ9KQJNzTg9fQgHdZYVIo5SudXkPEwur0KiJyq9TuLEvC+s6tvHyqg6nsGF9/7fdpibfz0S2/RKhy6WXizBGKP3oB8nnCXV20fepTpPOzREJR4ps2Ed9U7vB6+cBp77aWqNOriMh8UGEiS4JnHveueoLdp3+C5xlDM2cuFCUAPxr4Kx7Nl1s8smPDPLX/68zmp0nlZ/jgxs+wsrXcr+Ti4qPkl8gVs4DpMo6IyDxRYSJLxpr2TTTFWsiX8oQub92Ix8mHfMzBTLTImdFDFEKOkIWvepuxZx6tiXaKpYJGehURmScqTGTJiIQiNMVaiIQiRC8bBO2zd3yJ0oY0+bAjnJ/iiewE07kJupMrSMZarrg/MyMWjqtviYjIPFJhIktKMtZ81fdCDQ0kgES0ke7kcgBKqRTZQ4fwmpoItbQQaigPN++cU58SEZEqUGEiS1bJL133EkxxaIiZ554DILpqFS0f/jAAU9lxDCMZa76kr4qIiNwa3S4sS9J0dpLR1BCFUv6a65VmZi48Pz+GSbnTa45sMfueOXNEROTW6FtVlqTyZRjHbH7mkuWuUKAwMkL26FEAQi0txNauJdzVRai9vbzMC9HZ2E1LvO3CJIEiIjI/1AYtS1JjJEmmkCJfzJIv5YiGYjjnGP3P/xlK5QHYoitXEuvvJ9bf/57tQ15Yl3BERKpAf+7JkuR5IRoiSQBmczMXOrOGWt69A6d02QiwUL6Mc/kgayIiMn/0J58sWQ3RJOlCikIpR76UIxaOE25vh1KJUFsbeO+t26ey4/jOpzXeTjh0Y5MBiojI9akwkSXLM4/GaJLZ3DSzuWmioRhNjz121duAC6U8hVIeM08DqomIVIkKE1nSEpFG0vkURb9ArpglHklceK80PU326FFCySShtjYyyVBlmwbdjSMiUiUqTGRJO99qMpObYjY/TSwcv9BiUhwdJf3qqwBEVq8m98DtAJoXR0SkivRnnyx5iUgjIQtR8otki5kLy0uzsxeeF7taAEc0FCOsu3FERKpG37Cy5JkZjbFmprMTpPIzRNIFShMTFAYGiK5ZgwOyXS0Yai0REak2FSYiQDycIOXNUPKLTOzdA/uPAND0+ON4/X3kMqN4FtKEfSIiVaZLOSKUW02S0Wai4TjR2LutIqXJSdKFFHC+06sm7hMRqSa1mIhUxCMJ4pEEuY4p3IoVhFpa8Hq6yRWzgOkyjojIAlBhInKZ2Nq1xNauxTlHKj8D+Rli4YTGLhERWQAqTEQuUygVmJo8hzs1QKGlARJxGro6g44lIrIkqI+JyGWc88lnU6QKsxSGzuGGR4mEokHHEhFZEtRiInKZaDhGYjpPeO8J8H2it29Tp1cRkQWiwkTkCpq6V5C/5178mRkiy5YHHUdEZMlQYSJyBeGODsIdHUHHEBFZctTHRERERGqGChMRERGpGSpMREREpGaoMBEREZGaocJEREREaoYKExEREakZKkxERESkZqgwERERkZqhwkRERERqhgoTERERqRkqTERERKRmqDARERGRmqHCRERERGqGChMRERGpGSpMREREpGaoMBEREZGaYc65oDNcl5mNACersOtOYLQK+60lOsb6oGOsH0vhOHWM9aGax7jaOdd1pTcWRWFSLWa22zm3K+gc1aRjrA86xvqxFI5Tx1gfgjpGXcoRERGRmqHCRERERGrGUi9Mvhp0gAWgY6wPOsb6sRSOU8dYHwI5xiXdx0RERERqy1JvMREREZEaosJEREREasaSLEzM7MNmdtDMjpjZl4POM1/M7ISZ7TWzN8xsd2VZu5n90MwOVx7bgs55o8zsP5jZsJntu2jZVY/LzP5R5dweNLMPBZP6xlzlGP+JmQ1UzucbZvYLF723GI+xz8x+bGbvmNnbZvableV1cy6vcYx1cy7NLG5mr5jZm5Vj/N8qy+vpPF7tGOvmPJ5nZiEze93Mvld5Hfx5dM4tqX9ACDgKrAWiwJvA1qBzzdOxnQA6L1v2L4AvV55/GfjdoHPexHE9AtwJ7LvecQFbK+c0BqypnOtQ0Mdwk8f4T4B/cIV1F+sx9gJ3Vp43AYcqx1I35/Iax1g35xIwIFl5HgFeBu6rs/N4tWOsm/N4Ufa/D3wD+F7ldeDncSm2mNwDHHHOHXPO5YE/AZ4MOFM1PQl8rfL8a8Ang4tyc5xzzwPjly2+2nE9CfyJcy7nnDsOHKF8zmvaVY7xahbrMQ46516rPJ8B3gFWUEfn8hrHeDWL8Ridc2628jJS+eeor/N4tWO8mkV3jABmthL4KPCHFy0O/DwuxcJkBXD6otdnuPYXx2LigB+Y2R4z+1JlWY9zbhDKX5pAd2Dp5tfVjqvezu//aGZvVS71nG9SXfTHaGb9wE7Kf4nW5bm87Bihjs5lpfn/DWAY+KFzru7O41WOEeroPAK/B/xDwL9oWeDncSkWJnaFZfVyz/SDzrk7gY8Av2FmjwQdKAD1dH7/HbAO2AEMAv9nZfmiPkYzSwJ/DvyWc276WqteYdmiOM4rHGNdnUvnXMk5twNYCdxjZtuusXo9HWPdnEcz+xgw7JzbM9dNrrCsKse4FAuTM0DfRa9XAmcDyjKvnHNnK4/DwLcpN7MNmVkvQOVxOLiE8+pqx1U359c5N1T5cvSBP+DdZtNFe4xmFqH8C/vrzrm/qCyuq3N5pWOsx3MJ4JybBJ4DPkydncfzLj7GOjuPDwKfMLMTlLs0PG5mf0QNnMelWJi8CmwwszVmFgU+B3w34Ey3zMwazazp/HPgg8A+ysf2xcpqXwS+E0zCeXe14/ou8Dkzi5nZGmAD8EoA+W7Z+S+Hik9RPp+wSI/RzAz4/4B3nHP/6qK36uZcXu0Y6+lcmlmXmbVWnieA9wMHqK/zeMVjrKfz6Jz7R865lc65fsq/B591zn2BGjiP4WrstJY554pm9j8Cf035Dp3/4Jx7O+BY86EH+Hb5e5Ew8A3n3NNm9irwTTP7NeAU8NkAM94UM/tj4FGg08zOAL8DfIUrHJdz7m0z+yawHygCv+GcKwUS/AZc5RgfNbMdlJtLTwD/HSzeY6T8F9ovA3sr1+4B/hfq61xe7Rg/X0fnshf4mpmFKP9x+03n3PfM7EXq5zxe7Rj/Sx2dx6sJ/P9HDUkvIiIiNWMpXsoRERGRGqXCRERERGqGChMRERGpGSpMREREpGaoMBEREZGaocJEZIGYWcdFs5Keu2iW0kkz21+Fz/snZvYPbnCb2ass/09m9pmrvPd750cZNrPnrDKzdeX1LjN77kYyXCPbr5jZv5mPfV3nczZXzsvrZrZuAT7vUavM7HqV97vM7Olq5xCpFSpMRBaIc27MObejMsz1vwf+r8rzHVw6V8UVmVnNjTtkZu3AfZVJCM/rNrOPBJXpaipjUszFJ4HvOOd2OueOBpgDAOfcCDBoZg/OdxaRWqTCRKQ2hMzsD8zsbTP7QWW0yfMtEP+7mf0E+E0zu8vMfmLliRr/+qKho/+ume238uRif3LRfrdW9nHMzP7u+YVm9vfNbF/l329dHsbK/k1ln09x9ckfPwNc/tf8vwT+8RX2eUmLh5l9z8werTyfNbPfrRzXj8zsnotyf+Ki3fSZ2dNmdtDMfueifX3BzF6ptHT8v+d/+Vf2+0/N7GXg/svy7DCzlyo/s2+bWZuZ/QLwW8DfNrMfX7b+L5rZv6o8/00zO1Z5vs7MXqg8f6LS0rLXypO8xSrLT5jZ/1pZ77Nm9mEzO1B5/Tcu+oz3XdSq9rpVRnMG/ivw31zlHIjUFRUmIrVhA/BvnXO3AZPApy96r9U59z7gXwO/D3zGOXcX8B+Af15Z58vATufcduC/v2jbzcCHKM/p8TtmFjGzu4BfBe4F7gN+3cx2XpbnU8Am4Hbg14EHrpL7QeDyScBeBHJm9thcDryiEXiuclwzwD8DPlDJ8U8vWu8eyr+gd1D+Bb/LzLYAf5PyJJY7gBLv/hJvBPY55+51zr1w2Wf+Z+C3Kz+zvcDvOOf+indbsy7P/zzwcOX5w8CYma0AHgJ+amZx4D8Bf9M5dzvlEZj/zkXbZ51zD1EuMv4A+HhlP8suWucfUB5Rc0flvUxl+e6LPlukrqkwEakNx51zb1Se7wH6L3rvTyuPm4BtwA+tPNz5P6Y8kRbAW8DXzewLlIeLPu8p51zOOTdKeTKuHsq/SL/tnEs552aBv+C9v/QeAf64MmHZWeDZq+TuBUausPyfcYVWk2vI827Ly17gJ865QuV5/0Xr/bBySSxTyf0Q8ARwF/Bq5efyBLC2sn6J8oR6lzCzFsoF308qi75G+Zivyjl3DkhWWjH6gG9UtnkY+Cnl83PcOXfoKvs8fx43V9Y77MpDb//RRev8DPhXldatVufc+XM5DCy/Vj6ReqHCRKQ25C56XuLSeaxSlUcD3j7fT8U5d7tz7oOV9z4K/FvKv6D3XNQf5Ur7vdL05Vcyl/kqMkD8PRs692xl+X0XLS5y6XfOxdsV3LvzY/hUcldmcb34Z3F5Jkf5eL520c9lk3Pun1Tez87zfB4vUm5tOki5GHmY8iWin3H9n2vqoudX/Nk6574C/G0gAbxkZpsrb8V5t/VEpK6pMBFZPA4CXWZ2P0DlssxtZuYBfc65HwP/EGgFktfYz/PAJ82swcozUX+K8i/Zy9f5nJmFKv1YrnZZ5h1g/VXe++eVPOedAHaYmWdmfbw7ZfyN+ICZtVf64HySckHwDPAZM+uGcodcM1t9rZ0456aACTM731L0y8BPrrHJec9TvtzyPPA65Z9LrrK/A0C/mZ3/eVxtnweANfbuHT+fP/+Gma1zzu11zv0u5cs35wuTjbw7k61IXau5Xv4icmXOubyVb9n915VLEWHg94BDwB9Vlhnl/hGTZlf+A94595qZ/SfenbL8D51zr1+22reBxylfSjnE1X9pP0V5htU/vMLn/JWZXXyZ52fA8co+9wGvXfOAr+wF4L9QLoa+4ZzbDWBm/xj4QaVIKwC/AZy8zr6+CPx7M2sAjlFuCbmen1K+jPO8c65kZqcpFxo457Jm9qvAn1VarF6l3F/lEpX1vgQ8ZWajlWPaVnn7typ9c0qUZ3H9fmX5Y5R/1iJ1T7MLi8gtqdxZ8jHn3GTQWeqVmT0PPOmcmwg6i0i1qTARkVtiZvcCGefcW0FnqUdm1kX5jqP/GnQWkYWgwkRERERqhjq/ioiISM1QYSIiIiI1Q4WJiIiI1AwVJiIiIlIzVJiIiIhIzfj/AbSZTZBFQbxrAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 648x504 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig=plt.figure(figsize=(9,7))\n",
    "ax=fig.add_subplot(111)\n",
    " \n",
    "\n",
    "ax.plot(df.X,df.Tan,c='#93c47d',ls='--',label='Tan',linewidth=2.0)\n",
    "ax.plot(df.X,df.Relu,c='#ea9999', ls=':',label='Relu',linewidth=3.0)\n",
    "ax.plot(df.X, df.Logit, c = '#d9ead3',ls='-.',label='Logit',linewidth=2.0, alpha = 0.6)\n",
    "\n",
    "ax.set_xlabel('Threshold (Number of words)')\n",
    "ax.set_ylabel('Accuracy in %')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78b79673",
   "metadata": {},
   "source": [
    "As expected, we encounter the aforementioned behaviour. The continuous drop is due to the new words from each song leading to a rapid dictionary expansion for which the network cannot account via its learning. Once the (dynamic) dictionary expansion slows down, based on the size of the static dictionary (i.e. the total pool of tokenized words available in our data), the NN begins to transform noise into signal exceptionally well, to the point of overfitting shortly after.\n",
    "\n",
    "Therefore, we may deduce that there is a \"desirable range\", mostly 200-400, where our network performs at its best. We can build a strategy using an ensemble of our methods based on these findings in order to predict genres with as much accuracy as possible. You can find out more [here](https://www.mathcha.io/editor/4Qmgou4GfLpHld1wQoSWg9vYIGXngyxSlBw1GL) regarding the mathematical implications of the network's progress throughout the threshold markers.\n",
    "\n",
    "Lastly, we may be interested in testing some extreme cases. The following solver paired with an adaptive learning rate is extremely thorough, which may initially led us to believe that it could potentially outperform our previous findings. However, we are to find out that it never truly converges, despite allowing a ridiculous amount of iterations to loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8475a542",
   "metadata": {},
   "outputs": [],
   "source": [
    "## NOTE: Running locally ill-advised due to exceptional runtime.\n",
    "\n",
    "# clf_long = MLPClassifier(random_state=1, max_iter=10**4, activation='logistic',\\\n",
    "#                        solver='lbfgs', learning_rate='adaptive').fit(X_train, y_train)\n",
    "\n",
    "# clf_long.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "631c557a",
   "metadata": {},
   "source": [
    "The score obtained was nothing better than our previous findings. We recorded an accuracy of slightly below 71% on the chosen example.\n",
    "\n",
    "What happens if we let the same algorithm run for an extremely short amount, ignoring its need to converge?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "id": "df1b0be9",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\AnacondaIsHere\\lib\\site-packages\\sklearn\\neural_network\\_multilayer_perceptron.py:500: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.7069431920649234"
      ]
     },
     "execution_count": 224,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_short = MLPClassifier(random_state=1, max_iter=50, activation='logistic',\\\n",
    "                       solver='lbfgs', learning_rate='adaptive').fit(X_train, y_train)\n",
    "clf_short.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d2b1e49",
   "metadata": {},
   "source": [
    "Surprisingly, we lost very little accuracy as opposed to our previous algorithm. Indeed, it seems like the minimal performance gain does not justify running such a tedious algorithm, so it's best we retain our previous findings using the gridsearch.\n",
    "\n",
    "We will now run the algorithm one last time, just to get a full picture of the optimal parameters for selection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "id": "183a7c67",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf_layer = MLPClassifier(random_state=1, max_iter=200, activation='logistic',hidden_layer_sizes = (15,10,6,3,3,6,10,15),\n",
    "                       solver='adam', learning_rate='invscaling').fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "308ae392",
   "metadata": {},
   "source": [
    "The parameters are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "id": "0227f69a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'activation': 'logistic',\n",
       " 'alpha': 0.0001,\n",
       " 'batch_size': 'auto',\n",
       " 'beta_1': 0.9,\n",
       " 'beta_2': 0.999,\n",
       " 'early_stopping': False,\n",
       " 'epsilon': 1e-08,\n",
       " 'hidden_layer_sizes': (15, 10, 6, 3, 3, 6, 10, 15),\n",
       " 'learning_rate': 'invscaling',\n",
       " 'learning_rate_init': 0.001,\n",
       " 'max_fun': 15000,\n",
       " 'max_iter': 200,\n",
       " 'momentum': 0.9,\n",
       " 'n_iter_no_change': 10,\n",
       " 'nesterovs_momentum': True,\n",
       " 'power_t': 0.5,\n",
       " 'random_state': 1,\n",
       " 'shuffle': True,\n",
       " 'solver': 'adam',\n",
       " 'tol': 0.0001,\n",
       " 'validation_fraction': 0.1,\n",
       " 'verbose': False,\n",
       " 'warm_start': False}"
      ]
     },
     "execution_count": 226,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_layer.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "421ed00b",
   "metadata": {},
   "source": [
    "We conclude this section with a summary of the (practical) MLP behaviour recorded on our best estimators. These findings finalize our project, as we are ready to develop a product based on an ensemble of the methods & models explored."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c015bfe2",
   "metadata": {},
   "source": [
    "**MLP behaviour:** \n",
    "\n",
    "    - Underfits up until ~20 threshold. Expected accuracy 55-58% acc.\n",
    "\n",
    "    - Performance drop until ~ 140 threshold. Drop up to 38-40% acc.\n",
    "    \n",
    "    - Performance steady increase until ~200 threshold. Expected accuracy of 75%. Becomes the **relevant algorithm**.\n",
    "    \n",
    "    - Performance spike until 250-260 threshold. Begins overfitting ~ 260 threshold. Acc keeps increasing, slowly.\n",
    "    \n",
    "    - Reach ~90% acc for threshold 300.\n",
    "    \n",
    "    - Despite overfit, accuracy also increases until 380-390 threshold up to ~92%, from which point it starts decreasing.\n",
    "    \n",
    "    - Reach global recorded minimum (on a steady decrease) of ~ 88% acc at threshold 500."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e0759ca",
   "metadata": {},
   "source": [
    "**References**   ~ * denotes personal work bearing author rights\n",
    "\n",
    "\n",
    "[1. RNN pre-processing](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer)\n",
    "\n",
    "\n",
    "[2. RNN implementation on NL](https://towardsdatascience.com/recurrent-neural-networks-by-example-in-python-ffd204f99470)\n",
    "\n",
    "\n",
    "[3. CNN implementation](https://towardsdatascience.com/feed-forward-neural-networks-how-to-successfully-build-them-in-python-74503409d99a)\n",
    "\n",
    "\n",
    "[4. MLP - NN with sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html)\n",
    "\n",
    "\n",
    "[5. How to interpret NN performance relative to data - a short mathematical guide](https://www.mathcha.io/editor/4Qmgou4GfLpHld1wQoSWg9vYIGXngyxSlBw1GL) *"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}