GenderPredictions.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from Utils import *\n",
    "from classifier.CNNClassifier import CNNClassifier\n",
    "from classifier.Classifier import Classifier\n",
    "from classifier.ConstantClassifier import ConstantClassifier\n",
    "from classifier.LinearClassifier import LinearClassifier\n",
    "from classifier.RFClassifier import RFClassifier\n",
    "from classifier.SNNClassifier import SNNClassifier\n",
    "from Settings import *\n",
    "from sklearn.model_selection import train_test_split\n",
    "from main import run_for_classifier\n",
    "\n",
    "SAVE=True\n",
    "LOAD=False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preliminary informations\n",
    "##### Disclaimer : The neural networks results may vary a bit when re-training them due to some randomness in tensorflow-gpu / cuDNN.\n",
    "\n",
    "The models were trained on a computer with an i7-6700K, 16GB RAM and a GTX 1070.    \n",
    "The main libraries used are Keras with tensorflow backend, librosa, scikit-learn.    \n",
    "###### If you want to avoid retraining every classifier, set LOAD=True above. The RF with 100 estimators couldn't be pushed to the repository due to its file size though.\n",
    "\n",
    "Let's begin by explaining how to measure the performance of the classifiers. There are two main metrics :    \n",
    "- The sample accuracy : It's the accuracy for predicting the label of a sample of a file. For the CNN, it's a window of 10 samples.\n",
    "- The file accuracy : It's the accuracy for predicting the label of a file. The label is simply calculated by taking the most predicted label on all the samples of the file.\n",
    "\n",
    "The files were cut in samples of 20 MFCC features using the librosa library. This was mostly sufficient to allow the training on most of the classifiers. No preprocessing was done on the audio files or the MFCC features except normalizing them.    \n",
    "Some more preprocessing was done to allow the files to be fed to a CNN (cutting the files in windows).\n",
    "\n",
    "The files were separated in train set and test set using 80/20 proportions. There are 2703 files in total."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "features_with_label = files_to_features_with_labels(list_files(AUDIO_FILES_DIR))\n",
    "train_set, test_set = train_test_split(features_with_label, random_state=SEED, train_size=TRAIN_PERCENT, test_size=1-TRAIN_PERCENT)\n",
    "\n",
    "def test_classifier(classifier : Classifier) -> None:\n",
    "    \"\"\"\n",
    "    Wrapper for method run_for_classifier.\n",
    "    :param classifier: The classifier to test\n",
    "    \"\"\"\n",
    "    one_d = not isinstance(classifier, CNNClassifier)\n",
    "    run_for_classifier(classifier, test_set=test_set, train_set=train_set, save=SAVE, load=LOAD, one_d=one_d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Constant Classifier\n",
    "This classifier always predicts male."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished loading/creating features\n",
      "Using classifier ConstantClassifier\n",
      "Training ConstantClassifier\n",
      "Saved ConstantClassifier\n",
      "Predicting on files...\n",
      "Predicting on samples...\n",
      "Test accuracy - files : 0.4824399260628466\n",
      "Test accuracy - samples : 0.5065437497537723\n"
     ]
    }
   ],
   "source": [
    "test_classifier(ConstantClassifier())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us some information on the files as well as a baseline :     \n",
    "- The files are distributed nearly uniformly between male and female\n",
    "- When taking duration into account, the files are distributed even more closely half male / half female\n",
    "\n",
    "This means that it is not really needed to perform balancing between the train and test set, assuming we distribute the files randomly between them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear SVC\n",
    "This classifier is created using SVM with a linear kernel. It could serve as another simple baseline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished loading/creating features\n",
      "Using classifier LinearClassifier - C 1\n",
      "Training LinearClassifier - C 1\n",
      "Saved LinearClassifier - C 1\n",
      "Predicting on files...\n",
      "Predicting on samples...\n",
      "Test accuracy - files : 0.822550831792976\n",
      "Test accuracy - samples : 0.7052279084426585\n"
     ]
    }
   ],
   "source": [
    "test_classifier(LinearClassifier(c=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Here, the linear SVC returns quite mediocre results, having a file-accuracy of only 82%, and a worse sample accuracy of 70%.    \n",
    "Trying to increase the C parameter to 1000 makes the classifier not converge (or very slowly. When tested earlier, it hadn't converged in 10000 iterations).    \n",
    "It isn't surprising that a linear classifier would fail on this problem, as it is probably non-linear. Improving the preprocessing as well as doing features selection could improve the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished loading/creating features\n",
      "Using classifier LinearClassifier - C 1000\n",
      "Training LinearClassifier - C 1000\n",
      "Saved LinearClassifier - C 1000\n",
      "Predicting on files...\n",
      "Predicting on samples...\n",
      "Test accuracy - files : 0.5508317929759704\n",
      "Test accuracy - samples : 0.5846669030453453\n"
     ]
    }
   ],
   "source": [
    "test_classifier(LinearClassifier(c=1000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RandomForestClassifier\n",
    "Let's test RandomForestClassifiers, which are known to be quite accurate when features are well-defined and results are dependent on these features, which should be the case with the MFCC features and the speaker's gender."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished loading/creating features\n",
      "Using classifier RFClassifier - n_est 10 - max_depth None\n",
      "Training RFClassifier - n_est 10 - max_depth None\n",
      "Saved RFClassifier - n_est 10 - max_depth None\n",
      "Predicting on files...\n",
      "Predicting on samples...\n",
      "Test accuracy - files : 0.9981515711645101\n",
      "Test accuracy - samples : 0.8465665996927078\n"
     ]
    }
   ],
   "source": [
    "test_classifier(RFClassifier(n_estimators=10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we can already see that the files accuracy is nearly perfect (in fact, only 1 file is mispredicted). The sample accuracy could be better though.    \n",
    "Let's try to improve it by increasing the number of trees. We may also get a 100% file accuracy by doing this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished loading/creating features\n",
      "Using classifier RFClassifier - n_est 100 - max_depth None\n",
      "Training RFClassifier - n_est 100 - max_depth None\n",
      "Saved RFClassifier - n_est 100 - max_depth None\n",
      "Predicting on files...\n",
      "Predicting on samples...\n",
      "Test accuracy - files : 0.9981515711645101\n",
      "Test accuracy - samples : 0.8814324547925777\n"
     ]
    }
   ],
   "source": [
    "test_classifier(RFClassifier(n_estimators=100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It doesn't seem to improve much and isn't really worth it given the increase in memory and time consumption. It is better, as expected (averaging over more votes should lead to a better result), but not needed for the files accuracy.    \n",
    "Let's try with 1 and 5 trees to see if we can go lower than 10 while preserving the accuracy :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished loading/creating features\n",
      "Using classifier RFClassifier - n_est 5 - max_depth None\n",
      "Training RFClassifier - n_est 5 - max_depth None\n",
      "Saved RFClassifier - n_est 5 - max_depth None\n",
      "Predicting on files...\n",
      "Predicting on samples...\n",
      "Test accuracy - files : 0.9981515711645101\n",
      "Test accuracy - samples : 0.8261513611472245\n",
      "\n",
      "\n",
      "Finished loading/creating features\n",
      "Using classifier RFClassifier - n_est 1 - max_depth None\n",
      "Training RFClassifier - n_est 1 - max_depth None\n",
      "Saved RFClassifier - n_est 1 - max_depth None\n",
      "Predicting on files...\n",
      "Predicting on samples...\n",
      "Test accuracy - files : 0.9926062846580407\n",
      "Test accuracy - samples : 0.746507505023047\n"
     ]
    }
   ],
   "source": [
    "test_classifier(RFClassifier(n_estimators=5))\n",
    "print(\"\\n\")\n",
    "test_classifier(RFClassifier(n_estimators=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Strangely enough, we get a really good result with even only one tree. The sample accuracy decreases quite a bit obviously, but the file accuracy is still very good. This means that there are features which are very relevant to finding the speaker's gender.   \n",
    "With 5 trees, the file accuracy is the same as the one with 10 trees.    \n",
    "The RandomForest seems to be very well suited for this problem, as expected. What was less expected was that such a performance was obtained using a very small number of estimators.    \n",
    "RandomForest is therefore a very efficient and easy way to solve this problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Shallow Neural Network\n",
    "Let's try using a shallow neural network with only a single fully-connected hidden layer.    \n",
    "In theory, such a simple neural net should be able to approximate every function perfectly, but in practice, a deep neural network usually yields better results than a wide one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished loading/creating features\n",
      "Using classifier SNNClassifier - units 64\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 64)                1344      \n",
      "_________________________________________________________________\n",
      "p_re_lu_1 (PReLU)            (None, 64)                64        \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 65        \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 1)                 0         \n",
      "=================================================================\n",
      "Total params: 1,473\n",
      "Trainable params: 1,473\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Training SNNClassifier - units 64\n",
      "Train on 384480 samples, validate on 96121 samples\n",
      "Epoch 1/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.5344 - acc: 0.7202 - val_loss: 0.4788 - val_acc: 0.7653\n",
      "Epoch 2/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.4531 - acc: 0.7782 - val_loss: 0.4453 - val_acc: 0.7874\n",
      "Epoch 3/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.4263 - acc: 0.7934 - val_loss: 0.4241 - val_acc: 0.7987\n",
      "Epoch 4/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.4097 - acc: 0.8035 - val_loss: 0.4078 - val_acc: 0.8090\n",
      "Epoch 5/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3978 - acc: 0.8116 - val_loss: 0.4050 - val_acc: 0.8125\n",
      "Epoch 6/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3891 - acc: 0.8167 - val_loss: 0.3945 - val_acc: 0.8168\n",
      "Epoch 7/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3831 - acc: 0.8206 - val_loss: 0.3927 - val_acc: 0.8165\n",
      "Epoch 8/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3784 - acc: 0.8233 - val_loss: 0.3842 - val_acc: 0.8248\n",
      "Epoch 9/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3753 - acc: 0.8256 - val_loss: 0.3815 - val_acc: 0.8218\n",
      "Epoch 10/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3722 - acc: 0.8268 - val_loss: 0.3841 - val_acc: 0.8253\n",
      "Epoch 11/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3690 - acc: 0.8285 - val_loss: 0.3829 - val_acc: 0.8226\n",
      "Epoch 12/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3664 - acc: 0.8305 - val_loss: 0.3755 - val_acc: 0.8271\n",
      "Epoch 13/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3638 - acc: 0.8316 - val_loss: 0.3726 - val_acc: 0.8267\n",
      "Epoch 14/300\n",
      "384480/384480 [==============================] - 10s 27us/step - loss: 0.3612 - acc: 0.8329 - val_loss: 0.3746 - val_acc: 0.8299\n",
      "Epoch 15/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3590 - acc: 0.8345 - val_loss: 0.3676 - val_acc: 0.8326\n",
      "Epoch 16/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.3561 - acc: 0.8359 - val_loss: 0.3787 - val_acc: 0.8227\n",
      "Epoch 17/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.3538 - acc: 0.8376 - val_loss: 0.3717 - val_acc: 0.8269\n",
      "Epoch 18/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3528 - acc: 0.8378 - val_loss: 0.3673 - val_acc: 0.8341\n",
      "Epoch 19/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3500 - acc: 0.8391 - val_loss: 0.3561 - val_acc: 0.8401\n",
      "Epoch 20/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3480 - acc: 0.8402 - val_loss: 0.3584 - val_acc: 0.8404\n",
      "Epoch 21/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3463 - acc: 0.8414 - val_loss: 0.3564 - val_acc: 0.8394\n",
      "Epoch 22/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3446 - acc: 0.8416 - val_loss: 0.3548 - val_acc: 0.8399\n",
      "Epoch 23/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.3430 - acc: 0.8436 - val_loss: 0.3636 - val_acc: 0.8372\n",
      "Epoch 24/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.3408 - acc: 0.8441 - val_loss: 0.3523 - val_acc: 0.8417\n",
      "Epoch 25/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3401 - acc: 0.8445 - val_loss: 0.3625 - val_acc: 0.8356\n",
      "Epoch 26/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3381 - acc: 0.8462 - val_loss: 0.3521 - val_acc: 0.8414\n",
      "Epoch 27/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3378 - acc: 0.8460 - val_loss: 0.3614 - val_acc: 0.8380\n",
      "Epoch 28/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3358 - acc: 0.8470 - val_loss: 0.3451 - val_acc: 0.8463\n",
      "Epoch 29/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.3346 - acc: 0.8479 - val_loss: 0.3433 - val_acc: 0.8462\n",
      "Epoch 30/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3336 - acc: 0.8484 - val_loss: 0.3485 - val_acc: 0.8415\n",
      "Epoch 31/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.3329 - acc: 0.8491 - val_loss: 0.3542 - val_acc: 0.8385\n",
      "Epoch 32/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.3315 - acc: 0.8503 - val_loss: 0.3427 - val_acc: 0.8467\n",
      "Epoch 33/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3305 - acc: 0.8503 - val_loss: 0.3483 - val_acc: 0.8443\n",
      "Epoch 34/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3292 - acc: 0.8515 - val_loss: 0.3399 - val_acc: 0.8487\n",
      "Epoch 35/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3279 - acc: 0.8521 - val_loss: 0.3367 - val_acc: 0.8514\n",
      "Epoch 36/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3270 - acc: 0.8520 - val_loss: 0.3386 - val_acc: 0.8484\n",
      "Epoch 37/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3256 - acc: 0.8530 - val_loss: 0.3403 - val_acc: 0.8470\n",
      "Epoch 38/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3240 - acc: 0.8539 - val_loss: 0.3353 - val_acc: 0.8529\n",
      "Epoch 39/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3235 - acc: 0.8542 - val_loss: 0.3475 - val_acc: 0.8417\n",
      "Epoch 40/300\n",
      "384480/384480 [==============================] - 11s 30us/step - loss: 0.3222 - acc: 0.8549 - val_loss: 0.3347 - val_acc: 0.8510\n",
      "Epoch 41/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.3212 - acc: 0.8558 - val_loss: 0.3333 - val_acc: 0.8540\n",
      "Epoch 42/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3204 - acc: 0.8557 - val_loss: 0.3314 - val_acc: 0.8534\n",
      "Epoch 43/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3202 - acc: 0.8560 - val_loss: 0.3382 - val_acc: 0.8486\n",
      "Epoch 44/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3195 - acc: 0.8561 - val_loss: 0.3348 - val_acc: 0.8509\n",
      "Epoch 45/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3185 - acc: 0.8571 - val_loss: 0.3336 - val_acc: 0.8541\n",
      "Epoch 46/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3185 - acc: 0.8567 - val_loss: 0.3311 - val_acc: 0.8541\n",
      "Epoch 47/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3176 - acc: 0.8573 - val_loss: 0.3355 - val_acc: 0.8506\n",
      "Epoch 48/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3169 - acc: 0.8579 - val_loss: 0.3327 - val_acc: 0.8546\n",
      "Epoch 49/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3164 - acc: 0.8578 - val_loss: 0.3272 - val_acc: 0.8566\n",
      "Epoch 50/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.3157 - acc: 0.8584 - val_loss: 0.3332 - val_acc: 0.8514\n",
      "Epoch 51/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3158 - acc: 0.8588 - val_loss: 0.3297 - val_acc: 0.8544\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 52/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.3150 - acc: 0.8586 - val_loss: 0.3292 - val_acc: 0.8560\n",
      "Epoch 53/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3150 - acc: 0.8592 - val_loss: 0.3339 - val_acc: 0.8537\n",
      "Epoch 54/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3144 - acc: 0.8595 - val_loss: 0.3289 - val_acc: 0.8561\n",
      "Epoch 55/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3143 - acc: 0.8587 - val_loss: 0.3306 - val_acc: 0.8533\n",
      "Epoch 56/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3140 - acc: 0.8594 - val_loss: 0.3264 - val_acc: 0.8574\n",
      "Epoch 57/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.3131 - acc: 0.8596 - val_loss: 0.3228 - val_acc: 0.8584\n",
      "Epoch 58/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3131 - acc: 0.8597 - val_loss: 0.3286 - val_acc: 0.8569\n",
      "Epoch 59/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.3127 - acc: 0.8596 - val_loss: 0.3251 - val_acc: 0.8575\n",
      "Epoch 60/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3121 - acc: 0.8603 - val_loss: 0.3267 - val_acc: 0.8579\n",
      "Epoch 61/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.3121 - acc: 0.8608 - val_loss: 0.3222 - val_acc: 0.8596\n",
      "Epoch 62/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.3116 - acc: 0.8606 - val_loss: 0.3282 - val_acc: 0.8550\n",
      "Epoch 63/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3114 - acc: 0.8606 - val_loss: 0.3287 - val_acc: 0.8545\n",
      "Epoch 64/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3109 - acc: 0.8612 - val_loss: 0.3380 - val_acc: 0.8466\n",
      "Epoch 65/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3109 - acc: 0.8609 - val_loss: 0.3296 - val_acc: 0.8532\n",
      "Epoch 66/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3105 - acc: 0.8609 - val_loss: 0.3269 - val_acc: 0.8570\n",
      "Epoch 67/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3105 - acc: 0.8617 - val_loss: 0.3293 - val_acc: 0.8562\n",
      "Epoch 68/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3097 - acc: 0.8616 - val_loss: 0.3246 - val_acc: 0.8577\n",
      "Epoch 69/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3097 - acc: 0.8616 - val_loss: 0.3300 - val_acc: 0.8549\n",
      "Epoch 70/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3094 - acc: 0.8615 - val_loss: 0.3357 - val_acc: 0.8515\n",
      "Epoch 71/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3094 - acc: 0.8618 - val_loss: 0.3231 - val_acc: 0.8587\n",
      "\n",
      "Epoch 00071: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513.\n",
      "Epoch 72/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2998 - acc: 0.8674 - val_loss: 0.3184 - val_acc: 0.8607\n",
      "Epoch 73/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2995 - acc: 0.8677 - val_loss: 0.3186 - val_acc: 0.8624\n",
      "Epoch 74/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2995 - acc: 0.8675 - val_loss: 0.3176 - val_acc: 0.8618\n",
      "Epoch 75/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2994 - acc: 0.8677 - val_loss: 0.3197 - val_acc: 0.8622\n",
      "Epoch 76/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2993 - acc: 0.8679 - val_loss: 0.3177 - val_acc: 0.8627\n",
      "Epoch 77/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2992 - acc: 0.8680 - val_loss: 0.3184 - val_acc: 0.8613\n",
      "Epoch 78/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2991 - acc: 0.8682 - val_loss: 0.3172 - val_acc: 0.8619\n",
      "Epoch 79/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2991 - acc: 0.8678 - val_loss: 0.3169 - val_acc: 0.8633\n",
      "Epoch 80/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2991 - acc: 0.8679 - val_loss: 0.3188 - val_acc: 0.8609\n",
      "Epoch 81/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2991 - acc: 0.8680 - val_loss: 0.3173 - val_acc: 0.8627\n",
      "Epoch 82/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2989 - acc: 0.8680 - val_loss: 0.3175 - val_acc: 0.8617\n",
      "Epoch 83/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2989 - acc: 0.8678 - val_loss: 0.3181 - val_acc: 0.8624\n",
      "Epoch 84/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2989 - acc: 0.8680 - val_loss: 0.3187 - val_acc: 0.8621\n",
      "Epoch 85/300\n",
      "384480/384480 [==============================] - 10s 27us/step - loss: 0.2988 - acc: 0.8681 - val_loss: 0.3209 - val_acc: 0.8600\n",
      "Epoch 86/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2988 - acc: 0.8683 - val_loss: 0.3177 - val_acc: 0.8625\n",
      "Epoch 87/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2987 - acc: 0.8680 - val_loss: 0.3188 - val_acc: 0.8614\n",
      "Epoch 88/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2986 - acc: 0.8681 - val_loss: 0.3185 - val_acc: 0.8611\n",
      "Epoch 89/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2986 - acc: 0.8683 - val_loss: 0.3182 - val_acc: 0.8617\n",
      "\n",
      "Epoch 00089: ReduceLROnPlateau reducing learning rate to 1.0000000474974514e-05.\n",
      "Epoch 90/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2977 - acc: 0.8687 - val_loss: 0.3170 - val_acc: 0.8626\n",
      "Epoch 91/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.2975 - acc: 0.8689 - val_loss: 0.3168 - val_acc: 0.8630\n",
      "Epoch 92/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2975 - acc: 0.8688 - val_loss: 0.3167 - val_acc: 0.8629\n",
      "Epoch 93/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.2975 - acc: 0.8690 - val_loss: 0.3167 - val_acc: 0.8629\n",
      "Epoch 94/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2975 - acc: 0.8688 - val_loss: 0.3169 - val_acc: 0.8626\n",
      "Epoch 00094: early stopping\n",
      "Saved SNNClassifier - units 64\n",
      "Predicting on files...\n",
      "Predicting on samples...\n",
      "Test accuracy - files : 0.9981515711645101\n",
      "Test accuracy - samples : 0.8652956703305362\n"
     ]
    }
   ],
   "source": [
    "test_classifier(SNNClassifier(num_units=64, verbose=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We already get a nearly perfect accuracy for files, which is very impressive with only 64 units. This probably means that the function to approximate / the problem is not really complex, but still non-linear.    \n",
    "Let's try to improve the accuracy by increasing the number of units in the dense layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished loading/creating features\n",
      "Using classifier SNNClassifier - units 128\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_3 (Dense)              (None, 128)               2688      \n",
      "_________________________________________________________________\n",
      "p_re_lu_2 (PReLU)            (None, 128)               128       \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 1)                 129       \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 1)                 0         \n",
      "=================================================================\n",
      "Total params: 2,945\n",
      "Trainable params: 2,945\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Training SNNClassifier - units 128\n",
      "Train on 384480 samples, validate on 96121 samples\n",
      "Epoch 1/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.5385 - acc: 0.7174 - val_loss: 0.4875 - val_acc: 0.7571\n",
      "Epoch 2/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.4507 - acc: 0.7789 - val_loss: 0.4389 - val_acc: 0.7896\n",
      "Epoch 3/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.4187 - acc: 0.7983 - val_loss: 0.4206 - val_acc: 0.8024\n",
      "Epoch 4/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3996 - acc: 0.8104 - val_loss: 0.4057 - val_acc: 0.8092\n",
      "Epoch 5/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3857 - acc: 0.8185 - val_loss: 0.3852 - val_acc: 0.8234\n",
      "Epoch 6/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3753 - acc: 0.8249 - val_loss: 0.3880 - val_acc: 0.8191\n",
      "Epoch 7/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3670 - acc: 0.8297 - val_loss: 0.3717 - val_acc: 0.8308\n",
      "Epoch 8/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3604 - acc: 0.8335 - val_loss: 0.3676 - val_acc: 0.8311\n",
      "Epoch 9/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3552 - acc: 0.8361 - val_loss: 0.3646 - val_acc: 0.8336\n",
      "Epoch 10/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3506 - acc: 0.8386 - val_loss: 0.3528 - val_acc: 0.8398\n",
      "Epoch 11/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.3472 - acc: 0.8402 - val_loss: 0.3577 - val_acc: 0.8380\n",
      "Epoch 12/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3438 - acc: 0.8422 - val_loss: 0.3558 - val_acc: 0.8408\n",
      "Epoch 13/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3412 - acc: 0.8441 - val_loss: 0.3437 - val_acc: 0.8486\n",
      "Epoch 14/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3384 - acc: 0.8457 - val_loss: 0.3466 - val_acc: 0.8427\n",
      "Epoch 15/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3358 - acc: 0.8471 - val_loss: 0.3486 - val_acc: 0.8443\n",
      "Epoch 16/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3341 - acc: 0.8481 - val_loss: 0.3509 - val_acc: 0.8447\n",
      "Epoch 17/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3316 - acc: 0.8494 - val_loss: 0.3386 - val_acc: 0.8510\n",
      "Epoch 18/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3299 - acc: 0.8502 - val_loss: 0.3476 - val_acc: 0.8432\n",
      "Epoch 19/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.3278 - acc: 0.8517 - val_loss: 0.3433 - val_acc: 0.8442\n",
      "Epoch 20/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3265 - acc: 0.8522 - val_loss: 0.3376 - val_acc: 0.8520\n",
      "Epoch 21/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3246 - acc: 0.8530 - val_loss: 0.3332 - val_acc: 0.8544\n",
      "Epoch 22/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3236 - acc: 0.8536 - val_loss: 0.3402 - val_acc: 0.8459\n",
      "Epoch 23/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3217 - acc: 0.8550 - val_loss: 0.3450 - val_acc: 0.8423\n",
      "Epoch 24/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3202 - acc: 0.8556 - val_loss: 0.3298 - val_acc: 0.8539\n",
      "Epoch 25/300\n",
      "384480/384480 [==============================] - 10s 27us/step - loss: 0.3190 - acc: 0.8562 - val_loss: 0.3283 - val_acc: 0.8567\n",
      "Epoch 26/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3177 - acc: 0.8569 - val_loss: 0.3321 - val_acc: 0.8517\n",
      "Epoch 27/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3166 - acc: 0.8573 - val_loss: 0.3336 - val_acc: 0.8496\n",
      "Epoch 28/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3150 - acc: 0.8583 - val_loss: 0.3269 - val_acc: 0.8563\n",
      "Epoch 29/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.3138 - acc: 0.8591 - val_loss: 0.3371 - val_acc: 0.8495\n",
      "Epoch 30/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.3128 - acc: 0.8593 - val_loss: 0.3324 - val_acc: 0.8537\n",
      "Epoch 31/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3112 - acc: 0.8607 - val_loss: 0.3218 - val_acc: 0.8607\n",
      "Epoch 32/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.3105 - acc: 0.8607 - val_loss: 0.3178 - val_acc: 0.8618\n",
      "Epoch 33/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.3102 - acc: 0.8611 - val_loss: 0.3243 - val_acc: 0.8565\n",
      "Epoch 34/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3085 - acc: 0.8615 - val_loss: 0.3211 - val_acc: 0.8608\n",
      "Epoch 35/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.3075 - acc: 0.8624 - val_loss: 0.3226 - val_acc: 0.8575\n",
      "Epoch 36/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.3073 - acc: 0.8624 - val_loss: 0.3204 - val_acc: 0.8595\n",
      "Epoch 37/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3060 - acc: 0.8627 - val_loss: 0.3159 - val_acc: 0.8615\n",
      "Epoch 38/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3050 - acc: 0.8642 - val_loss: 0.3130 - val_acc: 0.8648\n",
      "Epoch 39/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3040 - acc: 0.8643 - val_loss: 0.3281 - val_acc: 0.8571\n",
      "Epoch 40/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3030 - acc: 0.8647 - val_loss: 0.3193 - val_acc: 0.8598\n",
      "Epoch 41/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.3026 - acc: 0.8649 - val_loss: 0.3215 - val_acc: 0.8592\n",
      "Epoch 42/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3020 - acc: 0.8649 - val_loss: 0.3160 - val_acc: 0.8615\n",
      "Epoch 43/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3010 - acc: 0.8659 - val_loss: 0.3198 - val_acc: 0.8585\n",
      "Epoch 44/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3005 - acc: 0.8656 - val_loss: 0.3111 - val_acc: 0.8642\n",
      "Epoch 45/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2988 - acc: 0.8669 - val_loss: 0.3203 - val_acc: 0.8574\n",
      "Epoch 46/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2986 - acc: 0.8673 - val_loss: 0.3194 - val_acc: 0.8585\n",
      "Epoch 47/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.2978 - acc: 0.8672 - val_loss: 0.3260 - val_acc: 0.8561\n",
      "Epoch 48/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2977 - acc: 0.8675 - val_loss: 0.3188 - val_acc: 0.8619\n",
      "Epoch 49/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2967 - acc: 0.8680 - val_loss: 0.3141 - val_acc: 0.8617\n",
      "Epoch 50/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2962 - acc: 0.8680 - val_loss: 0.3147 - val_acc: 0.8614\n",
      "Epoch 51/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2957 - acc: 0.8687 - val_loss: 0.3079 - val_acc: 0.8676\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 52/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2953 - acc: 0.8684 - val_loss: 0.3134 - val_acc: 0.8621\n",
      "Epoch 53/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2943 - acc: 0.8689 - val_loss: 0.3103 - val_acc: 0.8641\n",
      "Epoch 54/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2942 - acc: 0.8691 - val_loss: 0.3155 - val_acc: 0.8637\n",
      "Epoch 55/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.2938 - acc: 0.8694 - val_loss: 0.3257 - val_acc: 0.8564\n",
      "Epoch 56/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2934 - acc: 0.8702 - val_loss: 0.3169 - val_acc: 0.8619\n",
      "Epoch 57/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.2932 - acc: 0.8700 - val_loss: 0.3057 - val_acc: 0.8681\n",
      "Epoch 58/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.2925 - acc: 0.8704 - val_loss: 0.3155 - val_acc: 0.8610\n",
      "Epoch 59/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.2918 - acc: 0.8706 - val_loss: 0.3100 - val_acc: 0.8638\n",
      "Epoch 60/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.2916 - acc: 0.8711 - val_loss: 0.3114 - val_acc: 0.8651\n",
      "Epoch 61/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.2907 - acc: 0.8711 - val_loss: 0.3105 - val_acc: 0.8646\n",
      "Epoch 62/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2908 - acc: 0.8712 - val_loss: 0.3071 - val_acc: 0.8677\n",
      "Epoch 63/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.2909 - acc: 0.8709 - val_loss: 0.3088 - val_acc: 0.8644\n",
      "Epoch 64/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.2895 - acc: 0.8719 - val_loss: 0.3133 - val_acc: 0.8632\n",
      "Epoch 65/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2900 - acc: 0.8712 - val_loss: 0.3117 - val_acc: 0.8649\n",
      "Epoch 66/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2891 - acc: 0.8722 - val_loss: 0.3041 - val_acc: 0.8669\n",
      "Epoch 67/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2887 - acc: 0.8724 - val_loss: 0.3092 - val_acc: 0.8651\n",
      "Epoch 68/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2887 - acc: 0.8719 - val_loss: 0.3119 - val_acc: 0.8650\n",
      "Epoch 69/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2878 - acc: 0.8732 - val_loss: 0.3037 - val_acc: 0.8671\n",
      "Epoch 70/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2871 - acc: 0.8728 - val_loss: 0.3265 - val_acc: 0.8547\n",
      "Epoch 71/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2875 - acc: 0.8731 - val_loss: 0.3088 - val_acc: 0.8632\n",
      "Epoch 72/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2873 - acc: 0.8730 - val_loss: 0.3039 - val_acc: 0.8689\n",
      "Epoch 73/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.2866 - acc: 0.8737 - val_loss: 0.3073 - val_acc: 0.8660\n",
      "Epoch 74/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2862 - acc: 0.8735 - val_loss: 0.3040 - val_acc: 0.8686\n",
      "Epoch 75/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.2860 - acc: 0.8739 - val_loss: 0.3109 - val_acc: 0.8624\n",
      "Epoch 76/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2860 - acc: 0.8739 - val_loss: 0.3133 - val_acc: 0.8630\n",
      "Epoch 77/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2856 - acc: 0.8739 - val_loss: 0.3046 - val_acc: 0.8693\n",
      "Epoch 78/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2854 - acc: 0.8741 - val_loss: 0.3076 - val_acc: 0.8672\n",
      "Epoch 79/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2850 - acc: 0.8739 - val_loss: 0.3144 - val_acc: 0.8628\n",
      "\n",
      "Epoch 00079: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513.\n",
      "Epoch 80/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.2721 - acc: 0.8819 - val_loss: 0.2938 - val_acc: 0.8738\n",
      "Epoch 81/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2717 - acc: 0.8820 - val_loss: 0.2940 - val_acc: 0.8735\n",
      "Epoch 82/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2717 - acc: 0.8822 - val_loss: 0.2952 - val_acc: 0.8739\n",
      "Epoch 83/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2716 - acc: 0.8823 - val_loss: 0.2965 - val_acc: 0.8732\n",
      "Epoch 84/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2715 - acc: 0.8824 - val_loss: 0.2951 - val_acc: 0.8737\n",
      "Epoch 85/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2715 - acc: 0.8821 - val_loss: 0.2975 - val_acc: 0.8719\n",
      "Epoch 86/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2713 - acc: 0.8822 - val_loss: 0.2943 - val_acc: 0.8733\n",
      "Epoch 87/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2712 - acc: 0.8825 - val_loss: 0.2943 - val_acc: 0.8733\n",
      "Epoch 88/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2710 - acc: 0.8826 - val_loss: 0.2961 - val_acc: 0.8723\n",
      "Epoch 89/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2710 - acc: 0.8825 - val_loss: 0.2953 - val_acc: 0.8728\n",
      "Epoch 90/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2710 - acc: 0.8825 - val_loss: 0.2955 - val_acc: 0.8738\n",
      "\n",
      "Epoch 00090: ReduceLROnPlateau reducing learning rate to 1.0000000474974514e-05.\n",
      "Epoch 91/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2697 - acc: 0.8834 - val_loss: 0.2938 - val_acc: 0.8741\n",
      "Epoch 92/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2695 - acc: 0.8834 - val_loss: 0.2939 - val_acc: 0.8738\n",
      "Epoch 93/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2695 - acc: 0.8833 - val_loss: 0.2943 - val_acc: 0.8737\n",
      "Epoch 94/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.2695 - acc: 0.8833 - val_loss: 0.2940 - val_acc: 0.8739\n",
      "Epoch 95/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.2694 - acc: 0.8834 - val_loss: 0.2936 - val_acc: 0.8742\n",
      "Epoch 00095: early stopping\n",
      "Saved SNNClassifier - units 128\n",
      "Predicting on files...\n",
      "Predicting on samples...\n",
      "Test accuracy - files : 1.0\n",
      "Test accuracy - samples : 0.87788677461293\n"
     ]
    }
   ],
   "source": [
    "test_classifier(SNNClassifier(num_units=128, verbose=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the sample accuracy is ~1% better, and the file accuracy is now perfect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished loading/creating features\n",
      "Using classifier SNNClassifier - units 256\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_5 (Dense)              (None, 256)               5376      \n",
      "_________________________________________________________________\n",
      "p_re_lu_3 (PReLU)            (None, 256)               256       \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 1)                 257       \n",
      "_________________________________________________________________\n",
      "activation_3 (Activation)    (None, 1)                 0         \n",
      "=================================================================\n",
      "Total params: 5,889\n",
      "Trainable params: 5,889\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Training SNNClassifier - units 256\n",
      "Train on 384480 samples, validate on 96121 samples\n",
      "Epoch 1/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.5143 - acc: 0.7346 - val_loss: 0.4591 - val_acc: 0.7785\n",
      "Epoch 2/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.4298 - acc: 0.7914 - val_loss: 0.4312 - val_acc: 0.7883\n",
      "Epoch 3/300\n",
      "384480/384480 [==============================] - 9s 23us/step - loss: 0.3992 - acc: 0.8109 - val_loss: 0.3929 - val_acc: 0.8178\n",
      "Epoch 4/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3824 - acc: 0.8209 - val_loss: 0.3835 - val_acc: 0.8243\n",
      "Epoch 5/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3721 - acc: 0.8262 - val_loss: 0.3781 - val_acc: 0.8236\n",
      "Epoch 6/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3629 - acc: 0.8317 - val_loss: 0.3702 - val_acc: 0.8274\n",
      "Epoch 7/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.3568 - acc: 0.8343 - val_loss: 0.3686 - val_acc: 0.8289\n",
      "Epoch 8/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3511 - acc: 0.8371 - val_loss: 0.3578 - val_acc: 0.8329\n",
      "Epoch 9/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3460 - acc: 0.8397 - val_loss: 0.3491 - val_acc: 0.8443\n",
      "Epoch 10/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3411 - acc: 0.8436 - val_loss: 0.3544 - val_acc: 0.8402\n",
      "Epoch 11/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3370 - acc: 0.8451 - val_loss: 0.3390 - val_acc: 0.8496\n",
      "Epoch 12/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3329 - acc: 0.8477 - val_loss: 0.3347 - val_acc: 0.8490\n",
      "Epoch 13/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3300 - acc: 0.8495 - val_loss: 0.3323 - val_acc: 0.8532\n",
      "Epoch 14/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3259 - acc: 0.8521 - val_loss: 0.3315 - val_acc: 0.8498\n",
      "Epoch 15/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3240 - acc: 0.8529 - val_loss: 0.3255 - val_acc: 0.8564\n",
      "Epoch 16/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3209 - acc: 0.8545 - val_loss: 0.3280 - val_acc: 0.8540\n",
      "Epoch 17/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3195 - acc: 0.8554 - val_loss: 0.3199 - val_acc: 0.8603\n",
      "Epoch 18/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.3170 - acc: 0.8561 - val_loss: 0.3263 - val_acc: 0.8562\n",
      "Epoch 19/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.3146 - acc: 0.8573 - val_loss: 0.3260 - val_acc: 0.8583\n",
      "Epoch 20/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3129 - acc: 0.8591 - val_loss: 0.3231 - val_acc: 0.8577\n",
      "Epoch 21/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3115 - acc: 0.8590 - val_loss: 0.3157 - val_acc: 0.8610\n",
      "Epoch 22/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3089 - acc: 0.8610 - val_loss: 0.3268 - val_acc: 0.8533\n",
      "Epoch 23/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.3076 - acc: 0.8617 - val_loss: 0.3114 - val_acc: 0.8633\n",
      "Epoch 24/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3061 - acc: 0.8626 - val_loss: 0.3111 - val_acc: 0.8634\n",
      "Epoch 25/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.3043 - acc: 0.8637 - val_loss: 0.3405 - val_acc: 0.8446\n",
      "Epoch 26/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.3036 - acc: 0.8639 - val_loss: 0.3132 - val_acc: 0.8617\n",
      "Epoch 27/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.3021 - acc: 0.8642 - val_loss: 0.3106 - val_acc: 0.8665\n",
      "Epoch 28/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.3012 - acc: 0.8652 - val_loss: 0.3056 - val_acc: 0.8677\n",
      "Epoch 29/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2995 - acc: 0.8657 - val_loss: 0.3171 - val_acc: 0.8603\n",
      "Epoch 30/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.2986 - acc: 0.8661 - val_loss: 0.3096 - val_acc: 0.8645\n",
      "Epoch 31/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2966 - acc: 0.8672 - val_loss: 0.3064 - val_acc: 0.8643\n",
      "Epoch 32/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2959 - acc: 0.8677 - val_loss: 0.3104 - val_acc: 0.8638\n",
      "Epoch 33/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2946 - acc: 0.8684 - val_loss: 0.3188 - val_acc: 0.8595\n",
      "Epoch 34/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2929 - acc: 0.8696 - val_loss: 0.3010 - val_acc: 0.8682\n",
      "Epoch 35/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2926 - acc: 0.8694 - val_loss: 0.3156 - val_acc: 0.8634\n",
      "Epoch 36/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2919 - acc: 0.8699 - val_loss: 0.3006 - val_acc: 0.8706\n",
      "Epoch 37/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2909 - acc: 0.8700 - val_loss: 0.3106 - val_acc: 0.8614\n",
      "Epoch 38/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2897 - acc: 0.8710 - val_loss: 0.3012 - val_acc: 0.8696\n",
      "Epoch 39/300\n",
      "384480/384480 [==============================] - 9s 25us/step - loss: 0.2891 - acc: 0.8715 - val_loss: 0.3027 - val_acc: 0.8683\n",
      "Epoch 40/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2878 - acc: 0.8716 - val_loss: 0.3007 - val_acc: 0.8685\n",
      "Epoch 41/300\n",
      "384480/384480 [==============================] - 10s 27us/step - loss: 0.2873 - acc: 0.8724 - val_loss: 0.3023 - val_acc: 0.8686\n",
      "Epoch 42/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2855 - acc: 0.8732 - val_loss: 0.3014 - val_acc: 0.8679\n",
      "Epoch 43/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2856 - acc: 0.8734 - val_loss: 0.3005 - val_acc: 0.8696\n",
      "Epoch 44/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2845 - acc: 0.8737 - val_loss: 0.3060 - val_acc: 0.8656\n",
      "Epoch 45/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2836 - acc: 0.8746 - val_loss: 0.3033 - val_acc: 0.8684\n",
      "Epoch 46/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2834 - acc: 0.8749 - val_loss: 0.3174 - val_acc: 0.8633\n",
      "\n",
      "Epoch 00046: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513.\n",
      "Epoch 47/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2668 - acc: 0.8838 - val_loss: 0.2877 - val_acc: 0.8761\n",
      "Epoch 48/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2660 - acc: 0.8840 - val_loss: 0.2886 - val_acc: 0.8761\n",
      "Epoch 49/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2659 - acc: 0.8843 - val_loss: 0.2872 - val_acc: 0.8764\n",
      "Epoch 50/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2658 - acc: 0.8844 - val_loss: 0.2875 - val_acc: 0.8766\n",
      "Epoch 51/300\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2654 - acc: 0.8846 - val_loss: 0.2864 - val_acc: 0.8768\n",
      "Epoch 52/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2652 - acc: 0.8846 - val_loss: 0.2880 - val_acc: 0.8768\n",
      "Epoch 53/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2651 - acc: 0.8843 - val_loss: 0.2885 - val_acc: 0.8744\n",
      "Epoch 54/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2648 - acc: 0.8846 - val_loss: 0.2869 - val_acc: 0.8761\n",
      "Epoch 55/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2646 - acc: 0.8849 - val_loss: 0.2859 - val_acc: 0.8773\n",
      "Epoch 56/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2645 - acc: 0.8849 - val_loss: 0.2870 - val_acc: 0.8764\n",
      "Epoch 57/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2643 - acc: 0.8853 - val_loss: 0.2858 - val_acc: 0.8773\n",
      "Epoch 58/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2642 - acc: 0.8851 - val_loss: 0.2874 - val_acc: 0.8759\n",
      "Epoch 59/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2640 - acc: 0.8851 - val_loss: 0.2877 - val_acc: 0.8765\n",
      "Epoch 60/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2638 - acc: 0.8851 - val_loss: 0.2859 - val_acc: 0.8768\n",
      "Epoch 61/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2637 - acc: 0.8855 - val_loss: 0.2884 - val_acc: 0.8754\n",
      "Epoch 62/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2633 - acc: 0.8859 - val_loss: 0.2886 - val_acc: 0.8762\n",
      "Epoch 63/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2632 - acc: 0.8857 - val_loss: 0.2857 - val_acc: 0.8768\n",
      "Epoch 64/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2632 - acc: 0.8855 - val_loss: 0.2847 - val_acc: 0.8776\n",
      "Epoch 65/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2629 - acc: 0.8853 - val_loss: 0.2853 - val_acc: 0.8762\n",
      "Epoch 66/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2627 - acc: 0.8859 - val_loss: 0.2869 - val_acc: 0.8757\n",
      "Epoch 67/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2626 - acc: 0.8859 - val_loss: 0.2876 - val_acc: 0.8765\n",
      "Epoch 68/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2624 - acc: 0.8859 - val_loss: 0.2884 - val_acc: 0.8748\n",
      "Epoch 69/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2622 - acc: 0.8858 - val_loss: 0.2861 - val_acc: 0.8765\n",
      "Epoch 70/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2621 - acc: 0.8859 - val_loss: 0.2851 - val_acc: 0.8778\n",
      "Epoch 71/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2619 - acc: 0.8861 - val_loss: 0.2836 - val_acc: 0.8785\n",
      "Epoch 72/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2620 - acc: 0.8857 - val_loss: 0.2862 - val_acc: 0.8765\n",
      "Epoch 73/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2617 - acc: 0.8863 - val_loss: 0.2849 - val_acc: 0.8772\n",
      "Epoch 74/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2615 - acc: 0.8865 - val_loss: 0.2852 - val_acc: 0.8771\n",
      "Epoch 75/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2613 - acc: 0.8864 - val_loss: 0.2850 - val_acc: 0.8778\n",
      "Epoch 76/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2612 - acc: 0.8865 - val_loss: 0.2856 - val_acc: 0.8772\n",
      "Epoch 77/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2610 - acc: 0.8866 - val_loss: 0.2858 - val_acc: 0.8775\n",
      "Epoch 78/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2610 - acc: 0.8867 - val_loss: 0.2852 - val_acc: 0.8776\n",
      "Epoch 79/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2607 - acc: 0.8868 - val_loss: 0.2854 - val_acc: 0.8782\n",
      "Epoch 80/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2607 - acc: 0.8870 - val_loss: 0.2843 - val_acc: 0.8782\n",
      "Epoch 81/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2606 - acc: 0.8870 - val_loss: 0.2856 - val_acc: 0.8777\n",
      "\n",
      "Epoch 00081: ReduceLROnPlateau reducing learning rate to 1.0000000474974514e-05.\n",
      "Epoch 82/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2587 - acc: 0.8879 - val_loss: 0.2841 - val_acc: 0.8785\n",
      "Epoch 83/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2584 - acc: 0.8879 - val_loss: 0.2835 - val_acc: 0.8789\n",
      "Epoch 84/300\n",
      "384480/384480 [==============================] - 10s 26us/step - loss: 0.2583 - acc: 0.8880 - val_loss: 0.2833 - val_acc: 0.8790\n",
      "Epoch 85/300\n",
      "384480/384480 [==============================] - 10s 25us/step - loss: 0.2583 - acc: 0.8880 - val_loss: 0.2833 - val_acc: 0.8785\n",
      "Epoch 86/300\n",
      "384480/384480 [==============================] - 9s 24us/step - loss: 0.2583 - acc: 0.8880 - val_loss: 0.2837 - val_acc: 0.8781\n",
      "Epoch 00086: early stopping\n",
      "Saved SNNClassifier - units 256\n",
      "Predicting on files...\n",
      "Predicting on samples...\n",
      "Test accuracy - files : 1.0\n",
      "Test accuracy - samples : 0.8819524878855928\n"
     ]
    }
   ],
   "source": [
    "test_classifier(SNNClassifier(num_units=256, verbose=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We gain a very marginal increase in sample accuracy. File accuracy is still perfect. Increasing the model complexity further shouldn't yield any worthy increase in accuracy, as this model (and even the one with 128 units) is complex enough to accurately class the audio files.\n",
    "\n",
    "I decided to not add any Dropout or regularization to the network because, as we can see in the results, there is little to no overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolutional Neural Network\n",
    "The CNN Classifier was more difficult to implement.    \n",
    "First, the input is different compared to the other classifiers. We need to give identical 2D inputs to the network, but the number of samples per file varies a lot.    \n",
    "- The first attempt was padding the smaller files to the size of the biggest one by adding 0s features. Due to the difference in size, the network couldn't learn anything, as most of the data was empty (smallest file has 46 samples, biggest has around 1200).   \n",
    "- The next attempt was cutting the files into smaller windows of 46 samples (and potentially padding the last sample of the file). The next cell is the (reduced) output of a run using this strategy.    \n",
    "- The current one is to cut the files into windows of 10 samples. We'll see the result later.\n",
    "\n",
    "As the purpose of this classifier is to be a deep neural network, I decided to put two convolutional layers along with two max pooling, followed by two denses layers and an output layer. The whole network can be seen in the following cell.    \n",
    "Due to the number of parameters for this network (parameters in each layer, layout of the network, etc), most of these are heuristically chosen from a previous project on road segmentation. \n",
    "\n",
    "About design decisions :    \n",
    "- Batch normalization is used for faster and better training\n",
    "- Prelu activation is used instead of relu for potential better results\n",
    "- Glorot Normal initialization is used, but it seems that it is a matter of preference between uniform and normal and has no real impact on performance\n",
    "- Dropouts are used after max pooling and dense layers to reduce overfitting\n",
    "- Kernel regularizers are used on the convolution and dense layers to reduce overfitting\n",
    "- (Both dropouts and regularizers were added after checking that the network could learn perfectly the training set)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Finished loading/creating features\n",
    "Using classifier CNNClassifier\n",
    "_________________________________________________________________\n",
    "Layer (type)                 Output Shape              Param #   \n",
    "=================================================================\n",
    "batch_normalization_13 (Batc (None, 46, 20, 1)         4         \n",
    "_________________________________________________________________\n",
    "conv2d_9 (Conv2D)            (None, 44, 18, 32)        320       \n",
    "_________________________________________________________________\n",
    "p_re_lu_17 (PReLU)           (None, 44, 18, 32)        25344     \n",
    "_________________________________________________________________\n",
    "batch_normalization_14 (Batc (None, 44, 18, 32)        128       \n",
    "_________________________________________________________________\n",
    "max_pooling2d_9 (MaxPooling2 (None, 43, 17, 32)        0         \n",
    "_________________________________________________________________\n",
    "dropout_17 (Dropout)         (None, 43, 17, 32)        0         \n",
    "_________________________________________________________________\n",
    "conv2d_10 (Conv2D)           (None, 41, 15, 64)        18496     \n",
    "_________________________________________________________________\n",
    "p_re_lu_18 (PReLU)           (None, 41, 15, 64)        39360     \n",
    "_________________________________________________________________\n",
    "batch_normalization_15 (Batc (None, 41, 15, 64)        256       \n",
    "_________________________________________________________________\n",
    "max_pooling2d_10 (MaxPooling (None, 40, 14, 64)        0         \n",
    "_________________________________________________________________\n",
    "dropout_18 (Dropout)         (None, 40, 14, 64)        0         \n",
    "_________________________________________________________________\n",
    "flatten_5 (Flatten)          (None, 35840)             0         \n",
    "_________________________________________________________________\n",
    "dense_13 (Dense)             (None, 512)               18350592  \n",
    "_________________________________________________________________\n",
    "p_re_lu_19 (PReLU)           (None, 512)               512       \n",
    "_________________________________________________________________\n",
    "dropout_19 (Dropout)         (None, 512)               0         \n",
    "_________________________________________________________________\n",
    "dense_14 (Dense)             (None, 256)               131328    \n",
    "_________________________________________________________________\n",
    "p_re_lu_20 (PReLU)           (None, 256)               256       \n",
    "_________________________________________________________________\n",
    "dropout_20 (Dropout)         (None, 256)               0         \n",
    "_________________________________________________________________\n",
    "dense_15 (Dense)             (None, 1)                 257       \n",
    "_________________________________________________________________\n",
    "activation_5 (Activation)    (None, 1)                 0         \n",
    "=================================================================\n",
    "Total params: 18,566,853\n",
    "Trainable params: 18,566,659\n",
    "Non-trainable params: 194\n",
    "_________________________________________________________________\n",
    "None\n",
    "Training CNNClassifier\n",
    "Train on 9183 samples, validate on 2296 samples\n",
    "Epoch 1/200\n",
    "9183/9183 [==============================] - 5s 527us/step - loss: 14.4615 - acc: 0.5050 - val_loss: 10.9467 - val_acc: 0.4808\n",
    "Epoch 2/200\n",
    "9183/9183 [==============================] - 3s 371us/step - loss: 7.1822 - acc: 0.5513 - val_loss: 4.3108 - val_acc: 0.7522\n",
    "...\n",
    "Epoch 17/200\n",
    "9183/9183 [==============================] - 3s 375us/step - loss: 0.7153 - acc: 0.9245 - val_loss: 0.6791 - val_acc: 0.9299\n",
    "...\n",
    "Epoch 50/200\n",
    "9183/9183 [==============================] - 3s 371us/step - loss: 0.3034 - acc: 0.9543 - val_loss: 0.3069 - val_acc: 0.9525\n",
    "...\n",
    "Epoch 92/200\n",
    "9183/9183 [==============================] - 4s 411us/step - loss: 0.1006 - acc: 0.9808 - val_loss: 0.1658 - val_acc: 0.9612\n",
    "...\n",
    "Epoch 153/200\n",
    "9183/9183 [==============================] - 4s 390us/step - loss: 0.0760 - acc: 0.9857 - val_loss: 0.1658 - val_acc: 0.9564\n",
    "Epoch 154/200\n",
    "9183/9183 [==============================] - 3s 381us/step - loss: 0.0759 - acc: 0.9858 - val_loss: 0.1660 - val_acc: 0.9573\n",
    "\n",
    "Epoch 00154: early stopping\n",
    "Saved CNNClassifier\n",
    "Predicting on files...\n",
    "Predicting on samples...\n",
    "Test accuracy - files : 0.9796672828096118\n",
    "Test accuracy - samples : 0.9539430086149768\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the CNN got a really good score for both file and sample accuracy, but it strangely was not perfect like the SNN and is even a bit worse than the random forest. Moreover, the sample accuracy can not really be compared to the one obtained with the previous classifiers, as the CNN has more context (46 times more information than the other classifiers).  \n",
    "The next run obtains a perfect file score by reducing the window size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished loading/creating features\n",
      "Using classifier CNNClassifier\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "batch_normalization_4 (Batch (None, 10, 20, 1)         4         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 8, 18, 32)         320       \n",
      "_________________________________________________________________\n",
      "p_re_lu_5 (PReLU)            (None, 8, 18, 32)         4608      \n",
      "_________________________________________________________________\n",
      "batch_normalization_5 (Batch (None, 8, 18, 32)         128       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_3 (MaxPooling2 (None, 7, 17, 32)         0         \n",
      "_________________________________________________________________\n",
      "dropout_5 (Dropout)          (None, 7, 17, 32)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 5, 15, 64)         18496     \n",
      "_________________________________________________________________\n",
      "p_re_lu_6 (PReLU)            (None, 5, 15, 64)         4800      \n",
      "_________________________________________________________________\n",
      "batch_normalization_6 (Batch (None, 5, 15, 64)         256       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_4 (MaxPooling2 (None, 4, 14, 64)         0         \n",
      "_________________________________________________________________\n",
      "dropout_6 (Dropout)          (None, 4, 14, 64)         0         \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 3584)              0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 512)               1835520   \n",
      "_________________________________________________________________\n",
      "p_re_lu_7 (PReLU)            (None, 512)               512       \n",
      "_________________________________________________________________\n",
      "dropout_7 (Dropout)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 256)               131328    \n",
      "_________________________________________________________________\n",
      "p_re_lu_8 (PReLU)            (None, 256)               256       \n",
      "_________________________________________________________________\n",
      "dropout_8 (Dropout)          (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 1)                 257       \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 1)                 0         \n",
      "=================================================================\n",
      "Total params: 1,996,485\n",
      "Trainable params: 1,996,291\n",
      "Non-trainable params: 194\n",
      "_________________________________________________________________\n",
      "None\n",
      "Training CNNClassifier\n",
      "Train on 39254 samples, validate on 9814 samples\n",
      "Epoch 1/300\n",
      "39254/39254 [==============================] - 4s 105us/step - loss: 2.9799 - acc: 0.7376 - val_loss: 0.8558 - val_acc: 0.8285\n",
      "Epoch 2/300\n",
      "39254/39254 [==============================] - 3s 89us/step - loss: 0.7200 - acc: 0.8183 - val_loss: 0.5951 - val_acc: 0.8444\n",
      "Epoch 3/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.5944 - acc: 0.8362 - val_loss: 0.5537 - val_acc: 0.8576\n",
      "Epoch 4/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.5497 - acc: 0.8446 - val_loss: 0.4940 - val_acc: 0.8707\n",
      "Epoch 5/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.5304 - acc: 0.8515 - val_loss: 0.4882 - val_acc: 0.8702\n",
      "Epoch 6/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.5119 - acc: 0.8585 - val_loss: 0.4961 - val_acc: 0.8602\n",
      "Epoch 7/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4962 - acc: 0.8564 - val_loss: 0.4519 - val_acc: 0.8811\n",
      "Epoch 8/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4911 - acc: 0.8609 - val_loss: 0.4647 - val_acc: 0.8770\n",
      "Epoch 9/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4754 - acc: 0.8628 - val_loss: 0.4239 - val_acc: 0.8899\n",
      "Epoch 10/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4710 - acc: 0.8651 - val_loss: 0.4298 - val_acc: 0.8849\n",
      "Epoch 11/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4663 - acc: 0.8655 - val_loss: 0.4505 - val_acc: 0.8692\n",
      "Epoch 12/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.4597 - acc: 0.8660 - val_loss: 0.4252 - val_acc: 0.8811\n",
      "Epoch 13/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4595 - acc: 0.8667 - val_loss: 0.4089 - val_acc: 0.8914\n",
      "Epoch 14/300\n",
      "39254/39254 [==============================] - 3s 89us/step - loss: 0.4512 - acc: 0.8687 - val_loss: 0.4577 - val_acc: 0.8628\n",
      "Epoch 15/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.4429 - acc: 0.8714 - val_loss: 0.4019 - val_acc: 0.8888\n",
      "Epoch 16/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4321 - acc: 0.8711 - val_loss: 0.4025 - val_acc: 0.8866\n",
      "Epoch 17/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.4421 - acc: 0.8714 - val_loss: 0.4009 - val_acc: 0.8886\n",
      "Epoch 18/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4462 - acc: 0.8716 - val_loss: 0.4695 - val_acc: 0.8672\n",
      "Epoch 19/300\n",
      "39254/39254 [==============================] - 3s 89us/step - loss: 0.4562 - acc: 0.8698 - val_loss: 0.4039 - val_acc: 0.8868\n",
      "Epoch 20/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.4365 - acc: 0.8728 - val_loss: 0.4366 - val_acc: 0.8738\n",
      "Epoch 21/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4356 - acc: 0.8779 - val_loss: 0.4069 - val_acc: 0.8840\n",
      "Epoch 22/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.4417 - acc: 0.8742 - val_loss: 0.4133 - val_acc: 0.8827\n",
      "Epoch 23/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4356 - acc: 0.8742 - val_loss: 0.4023 - val_acc: 0.8826\n",
      "Epoch 24/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4344 - acc: 0.8754 - val_loss: 0.3996 - val_acc: 0.8891\n",
      "Epoch 25/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.4241 - acc: 0.8733 - val_loss: 0.3984 - val_acc: 0.8850\n",
      "Epoch 26/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.4414 - acc: 0.8745 - val_loss: 0.3923 - val_acc: 0.8881\n",
      "Epoch 27/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4272 - acc: 0.8767 - val_loss: 0.4000 - val_acc: 0.8866\n",
      "Epoch 28/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4388 - acc: 0.8752 - val_loss: 0.4214 - val_acc: 0.8838\n",
      "Epoch 29/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.4249 - acc: 0.8763 - val_loss: 0.3883 - val_acc: 0.8923\n",
      "Epoch 30/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.4281 - acc: 0.8747 - val_loss: 0.3888 - val_acc: 0.8947\n",
      "Epoch 31/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4227 - acc: 0.8774 - val_loss: 0.4029 - val_acc: 0.8821\n",
      "Epoch 32/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4232 - acc: 0.8769 - val_loss: 0.4010 - val_acc: 0.8887\n",
      "Epoch 33/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.4250 - acc: 0.8777 - val_loss: 0.4070 - val_acc: 0.8852\n",
      "Epoch 34/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.4218 - acc: 0.8763 - val_loss: 0.3949 - val_acc: 0.8881\n",
      "Epoch 35/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4182 - acc: 0.8765 - val_loss: 0.3870 - val_acc: 0.8935\n",
      "Epoch 36/300\n",
      "39254/39254 [==============================] - 4s 92us/step - loss: 0.4309 - acc: 0.8766 - val_loss: 0.4028 - val_acc: 0.8833\n",
      "Epoch 37/300\n",
      "39254/39254 [==============================] - 4s 91us/step - loss: 0.4212 - acc: 0.8781 - val_loss: 0.4029 - val_acc: 0.8936\n",
      "Epoch 38/300\n",
      "39254/39254 [==============================] - 3s 85us/step - loss: 0.4190 - acc: 0.8767 - val_loss: 0.3841 - val_acc: 0.8847\n",
      "Epoch 39/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.4189 - acc: 0.8800 - val_loss: 0.3985 - val_acc: 0.8813\n",
      "Epoch 40/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.4084 - acc: 0.8807 - val_loss: 0.3781 - val_acc: 0.8921\n",
      "Epoch 41/300\n",
      "39254/39254 [==============================] - 4s 89us/step - loss: 0.4282 - acc: 0.8789 - val_loss: 0.4001 - val_acc: 0.8857\n",
      "Epoch 42/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.4293 - acc: 0.8778 - val_loss: 0.3958 - val_acc: 0.8868\n",
      "Epoch 43/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.4188 - acc: 0.8822 - val_loss: 0.3724 - val_acc: 0.8933\n",
      "Epoch 44/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.4191 - acc: 0.8816 - val_loss: 0.3951 - val_acc: 0.8902\n",
      "Epoch 45/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4151 - acc: 0.8815 - val_loss: 0.3889 - val_acc: 0.8956\n",
      "Epoch 46/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.4137 - acc: 0.8787 - val_loss: 0.3878 - val_acc: 0.8955\n",
      "Epoch 47/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4116 - acc: 0.8791 - val_loss: 0.3783 - val_acc: 0.8940\n",
      "Epoch 48/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.4089 - acc: 0.8796 - val_loss: 0.3940 - val_acc: 0.8907\n",
      "Epoch 49/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.4075 - acc: 0.8817 - val_loss: 0.3784 - val_acc: 0.8957\n",
      "Epoch 50/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.4033 - acc: 0.8793 - val_loss: 0.3730 - val_acc: 0.8912\n",
      "Epoch 51/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.3972 - acc: 0.8810 - val_loss: 0.3885 - val_acc: 0.8807\n",
      "Epoch 52/300\n",
      "39254/39254 [==============================] - 3s 89us/step - loss: 0.3956 - acc: 0.8803 - val_loss: 0.3905 - val_acc: 0.8816\n",
      "Epoch 53/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.4045 - acc: 0.8818 - val_loss: 0.3639 - val_acc: 0.9006\n",
      "Epoch 54/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.4014 - acc: 0.8802 - val_loss: 0.3568 - val_acc: 0.8973\n",
      "Epoch 55/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.3905 - acc: 0.8814 - val_loss: 0.3663 - val_acc: 0.8886\n",
      "Epoch 56/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.3946 - acc: 0.8806 - val_loss: 0.3902 - val_acc: 0.8799\n",
      "Epoch 57/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.4005 - acc: 0.8818 - val_loss: 0.3871 - val_acc: 0.8905\n",
      "Epoch 58/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.4030 - acc: 0.8805 - val_loss: 0.3611 - val_acc: 0.8978\n",
      "Epoch 59/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.3886 - acc: 0.8845 - val_loss: 0.3760 - val_acc: 0.8923\n",
      "Epoch 60/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.4049 - acc: 0.8807 - val_loss: 0.3912 - val_acc: 0.8928\n",
      "Epoch 61/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.3962 - acc: 0.8843 - val_loss: 0.3684 - val_acc: 0.8872\n",
      "Epoch 62/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.3854 - acc: 0.8840 - val_loss: 0.3709 - val_acc: 0.8944\n",
      "Epoch 63/300\n",
      "39254/39254 [==============================] - 4s 89us/step - loss: 0.3943 - acc: 0.8826 - val_loss: 0.3675 - val_acc: 0.8908\n",
      "Epoch 64/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.3871 - acc: 0.8807 - val_loss: 0.3639 - val_acc: 0.8922\n",
      "\n",
      "Epoch 00064: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513.\n",
      "Epoch 65/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.3347 - acc: 0.8971 - val_loss: 0.3084 - val_acc: 0.9090\n",
      "Epoch 66/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.3041 - acc: 0.9040 - val_loss: 0.2925 - val_acc: 0.9131\n",
      "Epoch 67/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2894 - acc: 0.9072 - val_loss: 0.2850 - val_acc: 0.9077\n",
      "Epoch 68/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2818 - acc: 0.9087 - val_loss: 0.2801 - val_acc: 0.9124\n",
      "Epoch 69/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.2743 - acc: 0.9107 - val_loss: 0.2745 - val_acc: 0.9128\n",
      "Epoch 70/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.2687 - acc: 0.9111 - val_loss: 0.2737 - val_acc: 0.9123\n",
      "Epoch 71/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2658 - acc: 0.9121 - val_loss: 0.2722 - val_acc: 0.9109\n",
      "Epoch 72/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.2637 - acc: 0.9117 - val_loss: 0.2622 - val_acc: 0.9135\n",
      "Epoch 73/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2613 - acc: 0.9127 - val_loss: 0.2733 - val_acc: 0.9097\n",
      "Epoch 74/300\n",
      "39254/39254 [==============================] - 3s 89us/step - loss: 0.2570 - acc: 0.9140 - val_loss: 0.2597 - val_acc: 0.9169\n",
      "Epoch 75/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.2528 - acc: 0.9154 - val_loss: 0.2568 - val_acc: 0.9157\n",
      "Epoch 76/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2529 - acc: 0.9136 - val_loss: 0.2544 - val_acc: 0.9169\n",
      "Epoch 77/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.2508 - acc: 0.9160 - val_loss: 0.2566 - val_acc: 0.9139\n",
      "Epoch 78/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.2490 - acc: 0.9162 - val_loss: 0.2555 - val_acc: 0.9178\n",
      "Epoch 79/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.2492 - acc: 0.9163 - val_loss: 0.2500 - val_acc: 0.9185\n",
      "Epoch 80/300\n",
      "39254/39254 [==============================] - 3s 89us/step - loss: 0.2482 - acc: 0.9170 - val_loss: 0.2442 - val_acc: 0.9215\n",
      "Epoch 81/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2437 - acc: 0.9181 - val_loss: 0.2543 - val_acc: 0.9157\n",
      "Epoch 82/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2426 - acc: 0.9187 - val_loss: 0.2519 - val_acc: 0.9170\n",
      "Epoch 83/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2430 - acc: 0.9170 - val_loss: 0.2522 - val_acc: 0.9177\n",
      "Epoch 84/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2426 - acc: 0.9178 - val_loss: 0.2553 - val_acc: 0.9136\n",
      "Epoch 85/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.2438 - acc: 0.9166 - val_loss: 0.2538 - val_acc: 0.9124\n",
      "Epoch 86/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.2394 - acc: 0.9189 - val_loss: 0.2484 - val_acc: 0.9179\n",
      "Epoch 87/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2382 - acc: 0.9188 - val_loss: 0.2498 - val_acc: 0.9148\n",
      "Epoch 88/300\n",
      "39254/39254 [==============================] - 3s 89us/step - loss: 0.2403 - acc: 0.9181 - val_loss: 0.2448 - val_acc: 0.9166\n",
      "Epoch 89/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2370 - acc: 0.9193 - val_loss: 0.2487 - val_acc: 0.9175\n",
      "Epoch 90/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.2360 - acc: 0.9207 - val_loss: 0.2404 - val_acc: 0.9214\n",
      "Epoch 91/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2373 - acc: 0.9185 - val_loss: 0.2439 - val_acc: 0.9208\n",
      "Epoch 92/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.2375 - acc: 0.9193 - val_loss: 0.2464 - val_acc: 0.9175\n",
      "Epoch 93/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2403 - acc: 0.9175 - val_loss: 0.2419 - val_acc: 0.9202\n",
      "Epoch 94/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2379 - acc: 0.9219 - val_loss: 0.2400 - val_acc: 0.9231\n",
      "Epoch 95/300\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.2347 - acc: 0.9203 - val_loss: 0.2501 - val_acc: 0.9159\n",
      "Epoch 96/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2324 - acc: 0.9219 - val_loss: 0.2383 - val_acc: 0.9213\n",
      "Epoch 97/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.2349 - acc: 0.9221 - val_loss: 0.2432 - val_acc: 0.9174\n",
      "Epoch 98/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.2321 - acc: 0.9213 - val_loss: 0.2398 - val_acc: 0.9228\n",
      "Epoch 99/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.2320 - acc: 0.9217 - val_loss: 0.2424 - val_acc: 0.9196\n",
      "Epoch 100/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2323 - acc: 0.9228 - val_loss: 0.2389 - val_acc: 0.9219\n",
      "Epoch 101/300\n",
      "39254/39254 [==============================] - 3s 87us/step - loss: 0.2317 - acc: 0.9227 - val_loss: 0.2431 - val_acc: 0.9214\n",
      "Epoch 102/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.2295 - acc: 0.9228 - val_loss: 0.2428 - val_acc: 0.9202\n",
      "Epoch 103/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.2312 - acc: 0.9231 - val_loss: 0.2383 - val_acc: 0.9229\n",
      "Epoch 104/300\n",
      "39254/39254 [==============================] - 3s 85us/step - loss: 0.2288 - acc: 0.9232 - val_loss: 0.2385 - val_acc: 0.9221\n",
      "Epoch 105/300\n",
      "39254/39254 [==============================] - 3s 85us/step - loss: 0.2281 - acc: 0.9230 - val_loss: 0.2376 - val_acc: 0.9196\n",
      "Epoch 106/300\n",
      "39254/39254 [==============================] - 4s 89us/step - loss: 0.2291 - acc: 0.9242 - val_loss: 0.2385 - val_acc: 0.9241\n",
      "Epoch 107/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.2275 - acc: 0.9238 - val_loss: 0.2414 - val_acc: 0.9198\n",
      "Epoch 108/300\n",
      "39254/39254 [==============================] - 3s 85us/step - loss: 0.2258 - acc: 0.9263 - val_loss: 0.2379 - val_acc: 0.9224\n",
      "Epoch 109/300\n",
      "39254/39254 [==============================] - 3s 85us/step - loss: 0.2294 - acc: 0.9238 - val_loss: 0.2347 - val_acc: 0.9255\n",
      "Epoch 110/300\n",
      "39254/39254 [==============================] - 3s 85us/step - loss: 0.2265 - acc: 0.9245 - val_loss: 0.2313 - val_acc: 0.9251\n",
      "Epoch 111/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.2291 - acc: 0.9220 - val_loss: 0.2356 - val_acc: 0.9199\n",
      "Epoch 112/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.2267 - acc: 0.9250 - val_loss: 0.2351 - val_acc: 0.9254\n",
      "Epoch 113/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.2303 - acc: 0.9241 - val_loss: 0.2344 - val_acc: 0.9227\n",
      "Epoch 114/300\n",
      "39254/39254 [==============================] - 3s 85us/step - loss: 0.2238 - acc: 0.9246 - val_loss: 0.2453 - val_acc: 0.9182\n",
      "Epoch 115/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.2246 - acc: 0.9251 - val_loss: 0.2428 - val_acc: 0.9222\n",
      "Epoch 116/300\n",
      "39254/39254 [==============================] - 3s 85us/step - loss: 0.2268 - acc: 0.9256 - val_loss: 0.2342 - val_acc: 0.9247\n",
      "Epoch 117/300\n",
      "39254/39254 [==============================] - 3s 85us/step - loss: 0.2239 - acc: 0.9248 - val_loss: 0.2430 - val_acc: 0.9181\n",
      "Epoch 118/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.2249 - acc: 0.9252 - val_loss: 0.2409 - val_acc: 0.9214\n",
      "Epoch 119/300\n",
      "39254/39254 [==============================] - 3s 85us/step - loss: 0.2257 - acc: 0.9267 - val_loss: 0.2389 - val_acc: 0.9231\n",
      "Epoch 120/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.2238 - acc: 0.9260 - val_loss: 0.2306 - val_acc: 0.9264\n",
      "Epoch 121/300\n",
      "39254/39254 [==============================] - 3s 85us/step - loss: 0.2235 - acc: 0.9248 - val_loss: 0.2370 - val_acc: 0.9211\n",
      "Epoch 122/300\n",
      "39254/39254 [==============================] - 3s 85us/step - loss: 0.2236 - acc: 0.9257 - val_loss: 0.2346 - val_acc: 0.9223\n",
      "Epoch 123/300\n",
      "39254/39254 [==============================] - 3s 86us/step - loss: 0.2240 - acc: 0.9241 - val_loss: 0.2441 - val_acc: 0.9204\n",
      "Epoch 124/300\n",
      "39254/39254 [==============================] - 3s 89us/step - loss: 0.2221 - acc: 0.9267 - val_loss: 0.2334 - val_acc: 0.9214\n",
      "Epoch 125/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.2246 - acc: 0.9260 - val_loss: 0.2338 - val_acc: 0.9258\n",
      "Epoch 126/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.2221 - acc: 0.9240 - val_loss: 0.2340 - val_acc: 0.9224\n",
      "Epoch 127/300\n",
      "39254/39254 [==============================] - 4s 92us/step - loss: 0.2226 - acc: 0.9267 - val_loss: 0.2326 - val_acc: 0.9240\n",
      "Epoch 128/300\n",
      "39254/39254 [==============================] - 4s 92us/step - loss: 0.2230 - acc: 0.9260 - val_loss: 0.2334 - val_acc: 0.9242\n",
      "Epoch 129/300\n",
      "39254/39254 [==============================] - 4s 92us/step - loss: 0.2202 - acc: 0.9283 - val_loss: 0.2392 - val_acc: 0.9179\n",
      "Epoch 130/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.2192 - acc: 0.9277 - val_loss: 0.2388 - val_acc: 0.9237\n",
      "\n",
      "Epoch 00130: ReduceLROnPlateau reducing learning rate to 1.0000000474974514e-05.\n",
      "Epoch 131/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.2150 - acc: 0.9297 - val_loss: 0.2286 - val_acc: 0.9275\n",
      "Epoch 132/300\n",
      "39254/39254 [==============================] - 4s 92us/step - loss: 0.2099 - acc: 0.9329 - val_loss: 0.2299 - val_acc: 0.9255\n",
      "Epoch 133/300\n",
      "39254/39254 [==============================] - 4s 97us/step - loss: 0.2088 - acc: 0.9332 - val_loss: 0.2332 - val_acc: 0.9236\n",
      "Epoch 134/300\n",
      "39254/39254 [==============================] - 4s 99us/step - loss: 0.2056 - acc: 0.9338 - val_loss: 0.2253 - val_acc: 0.9271\n",
      "Epoch 135/300\n",
      "39254/39254 [==============================] - 4s 94us/step - loss: 0.2053 - acc: 0.9329 - val_loss: 0.2260 - val_acc: 0.9295\n",
      "Epoch 136/300\n",
      "39254/39254 [==============================] - 4s 91us/step - loss: 0.2056 - acc: 0.9334 - val_loss: 0.2289 - val_acc: 0.9262\n",
      "Epoch 137/300\n",
      "39254/39254 [==============================] - 4s 94us/step - loss: 0.2030 - acc: 0.9345 - val_loss: 0.2257 - val_acc: 0.9258\n",
      "Epoch 138/300\n",
      "39254/39254 [==============================] - 4s 92us/step - loss: 0.2023 - acc: 0.9352 - val_loss: 0.2284 - val_acc: 0.9258\n",
      "Epoch 139/300\n",
      "39254/39254 [==============================] - 4s 93us/step - loss: 0.2032 - acc: 0.9325 - val_loss: 0.2247 - val_acc: 0.9275\n",
      "Epoch 140/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.2001 - acc: 0.9350 - val_loss: 0.2235 - val_acc: 0.9301\n",
      "Epoch 141/300\n",
      "39254/39254 [==============================] - 4s 91us/step - loss: 0.2016 - acc: 0.9344 - val_loss: 0.2246 - val_acc: 0.9278\n",
      "Epoch 142/300\n",
      "39254/39254 [==============================] - 4s 91us/step - loss: 0.1988 - acc: 0.9337 - val_loss: 0.2224 - val_acc: 0.9259\n",
      "Epoch 143/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.2004 - acc: 0.9339 - val_loss: 0.2227 - val_acc: 0.9287\n",
      "Epoch 144/300\n",
      "39254/39254 [==============================] - 4s 93us/step - loss: 0.1998 - acc: 0.9347 - val_loss: 0.2236 - val_acc: 0.9291\n",
      "Epoch 145/300\n",
      "39254/39254 [==============================] - 4s 91us/step - loss: 0.2015 - acc: 0.9342 - val_loss: 0.2231 - val_acc: 0.9270\n",
      "Epoch 146/300\n",
      "39254/39254 [==============================] - 4s 91us/step - loss: 0.1983 - acc: 0.9346 - val_loss: 0.2226 - val_acc: 0.9288\n",
      "Epoch 147/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.1991 - acc: 0.9347 - val_loss: 0.2236 - val_acc: 0.9269\n",
      "Epoch 148/300\n",
      "39254/39254 [==============================] - 4s 91us/step - loss: 0.1990 - acc: 0.9359 - val_loss: 0.2187 - val_acc: 0.9307\n",
      "Epoch 149/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.1964 - acc: 0.9370 - val_loss: 0.2271 - val_acc: 0.9248\n",
      "Epoch 150/300\n",
      "39254/39254 [==============================] - 4s 91us/step - loss: 0.1967 - acc: 0.9360 - val_loss: 0.2232 - val_acc: 0.9286\n",
      "Epoch 151/300\n",
      "39254/39254 [==============================] - 4s 89us/step - loss: 0.1965 - acc: 0.9362 - val_loss: 0.2216 - val_acc: 0.9266\n",
      "Epoch 152/300\n",
      "39254/39254 [==============================] - 4s 92us/step - loss: 0.1970 - acc: 0.9358 - val_loss: 0.2251 - val_acc: 0.9260\n",
      "Epoch 153/300\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "39254/39254 [==============================] - 4s 93us/step - loss: 0.1957 - acc: 0.9363 - val_loss: 0.2166 - val_acc: 0.9306\n",
      "Epoch 154/300\n",
      "39254/39254 [==============================] - 3s 88us/step - loss: 0.1959 - acc: 0.9374 - val_loss: 0.2172 - val_acc: 0.9310\n",
      "Epoch 155/300\n",
      "39254/39254 [==============================] - 4s 91us/step - loss: 0.1960 - acc: 0.9349 - val_loss: 0.2189 - val_acc: 0.9288\n",
      "Epoch 156/300\n",
      "39254/39254 [==============================] - 4s 91us/step - loss: 0.1956 - acc: 0.9368 - val_loss: 0.2230 - val_acc: 0.9269\n",
      "Epoch 157/300\n",
      "39254/39254 [==============================] - 4s 91us/step - loss: 0.1942 - acc: 0.9372 - val_loss: 0.2202 - val_acc: 0.9300\n",
      "Epoch 158/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.1969 - acc: 0.9347 - val_loss: 0.2170 - val_acc: 0.9282\n",
      "Epoch 159/300\n",
      "39254/39254 [==============================] - 4s 92us/step - loss: 0.1923 - acc: 0.9376 - val_loss: 0.2167 - val_acc: 0.9310\n",
      "Epoch 160/300\n",
      "39254/39254 [==============================] - 4s 92us/step - loss: 0.1930 - acc: 0.9371 - val_loss: 0.2240 - val_acc: 0.9281\n",
      "Epoch 161/300\n",
      "39254/39254 [==============================] - 3s 89us/step - loss: 0.1926 - acc: 0.9385 - val_loss: 0.2202 - val_acc: 0.9287\n",
      "Epoch 162/300\n",
      "39254/39254 [==============================] - 4s 92us/step - loss: 0.1912 - acc: 0.9366 - val_loss: 0.2175 - val_acc: 0.9285\n",
      "Epoch 163/300\n",
      "39254/39254 [==============================] - 4s 94us/step - loss: 0.1929 - acc: 0.9372 - val_loss: 0.2213 - val_acc: 0.9263\n",
      "\n",
      "Epoch 00163: ReduceLROnPlateau reducing learning rate to 1.0000000656873453e-06.\n",
      "Epoch 164/300\n",
      "39254/39254 [==============================] - 4s 91us/step - loss: 0.1942 - acc: 0.9360 - val_loss: 0.2154 - val_acc: 0.9300\n",
      "Epoch 165/300\n",
      "39254/39254 [==============================] - 4s 92us/step - loss: 0.1929 - acc: 0.9374 - val_loss: 0.2179 - val_acc: 0.9286\n",
      "Epoch 166/300\n",
      "39254/39254 [==============================] - 4s 91us/step - loss: 0.1917 - acc: 0.9363 - val_loss: 0.2163 - val_acc: 0.9310\n",
      "Epoch 167/300\n",
      "39254/39254 [==============================] - 4s 93us/step - loss: 0.1939 - acc: 0.9364 - val_loss: 0.2172 - val_acc: 0.9292\n",
      "Epoch 168/300\n",
      "39254/39254 [==============================] - 4s 91us/step - loss: 0.1937 - acc: 0.9356 - val_loss: 0.2161 - val_acc: 0.9305\n",
      "Epoch 169/300\n",
      "39254/39254 [==============================] - 4s 92us/step - loss: 0.1914 - acc: 0.9376 - val_loss: 0.2200 - val_acc: 0.9280\n",
      "Epoch 170/300\n",
      "39254/39254 [==============================] - 4s 93us/step - loss: 0.1905 - acc: 0.9361 - val_loss: 0.2178 - val_acc: 0.9289\n",
      "Epoch 171/300\n",
      "39254/39254 [==============================] - 4s 92us/step - loss: 0.1912 - acc: 0.9391 - val_loss: 0.2156 - val_acc: 0.9298\n",
      "Epoch 172/300\n",
      "39254/39254 [==============================] - 4s 93us/step - loss: 0.1907 - acc: 0.9380 - val_loss: 0.2153 - val_acc: 0.9311\n",
      "Epoch 173/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.1926 - acc: 0.9365 - val_loss: 0.2219 - val_acc: 0.9278\n",
      "Epoch 174/300\n",
      "39254/39254 [==============================] - 4s 93us/step - loss: 0.1905 - acc: 0.9392 - val_loss: 0.2222 - val_acc: 0.9278\n",
      "Epoch 175/300\n",
      "39254/39254 [==============================] - 4s 92us/step - loss: 0.1896 - acc: 0.9394 - val_loss: 0.2239 - val_acc: 0.9257\n",
      "Epoch 176/300\n",
      "39254/39254 [==============================] - 4s 90us/step - loss: 0.1941 - acc: 0.9356 - val_loss: 0.2204 - val_acc: 0.9276\n",
      "Epoch 177/300\n",
      "39254/39254 [==============================] - 4s 93us/step - loss: 0.1905 - acc: 0.9381 - val_loss: 0.2221 - val_acc: 0.9270\n",
      "Epoch 178/300\n",
      "39254/39254 [==============================] - 4s 89us/step - loss: 0.1904 - acc: 0.9382 - val_loss: 0.2167 - val_acc: 0.9306\n",
      "Epoch 179/300\n",
      "39254/39254 [==============================] - 4s 93us/step - loss: 0.1939 - acc: 0.9377 - val_loss: 0.2169 - val_acc: 0.9299\n",
      "Epoch 00179: early stopping\n",
      "Saved CNNClassifier\n",
      "Predicting on files...\n",
      "Predicting on samples...\n",
      "Test accuracy - files : 0.9981515711645101\n",
      "Test accuracy - samples : 0.9319990727146279\n"
     ]
    }
   ],
   "source": [
    "test_classifier(CNNClassifier(verbose=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, by using windows of size 10, we get a the same result as the RandomForest for file accuracy (1 mispredicted file), which is better than with a window size of 46. It is probably mainly due to averaging over more samples : As the per sample accuracy is still very high, averaging over them allows for a very good score. In theory, as long as we have a per sample accuracy >50% and a very high number of samples per file, the predictions should be perfect. It is not always the case in practice due to the number of samples per file. \n",
    "\n",
    "\n",
    "100% files accuracy should be obtainable by tuning the parameters or changing the layout.      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compared to the other classifiers however, it is slower to train, more memory hungry, and therefore seems a bit overkill for this task, given than a simple neural net with a single hidden layer obtains file predictions results better than this one.    \n",
    "\n",
    "### Dumps size\n",
    "If we compare the performance/file size metric of the classifiers, we see that the SNN is easily the best one, the 256 units network weighing only 96Ko, compared to the 12Mo for the 10 estimators RandomForest or the 23Mo CNN. The linear SVC only weighs 1Ko, but gives bad results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other considerations\n",
    "- I've also added a cross-validation method to test the classifiers on samples, but due to the time it takes to execute it, I decided to leave it out.\n",
    "- I have considered data augmentation, but given the results we've already obtained, this isn't at all necessary for this problem and would only be wasted time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "In conclusion, we can rate the classifiers like this : \n",
    "- The winner is obviously the SNN. The complexity of designing it is really low, the file size is also very small, and it gets a perfect result.\n",
    "- The second one would be the RandomForest. The design complexity is non-existant, the file size is moderately high but still very manageable for a low number of trees, and it gets a nearly perfect result.\n",
    "- The third is the CNN. Although the design complexity is high due to the number of parameters we can change and layouts we can do, and the file size is higher than the others, it gets a nearly score. It is totally unnecessary to implement such a complex network for this problem though.\n",
    "- The fourth is the linear SVC, because even though the design complexity and file size is null, the results are bad.\n",
    "- For obvious reasons, the constant classifier is the last one.\n",
    "\n",
    "In short, a more complex model is not always better than a simple one. If the neural network doesn't have to learn complex features, which is the case here as it seems that the MFCC features are expressive enough, a simple model can do a very good job while being faster and less resource-intensive to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}