1.FeedForward.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FeedForward 🥸"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we explore the possible use of Multilayer Perceptron Neural Network in this classicification task. We used three different architecture strucutures: a baseline Network with two hidden layers and 1100 hidden neurons and then two variations obtained by doubleing the number of neurons or layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Libriaries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just import all the usefull libriaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import utilities as ff\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "from keras.layers import *\n",
    "from keras.models import *\n",
    "from keras.losses import *\n",
    "from keras.optimizers import *\n",
    "from keras.utils import *\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img_size = 100\n",
    "color_mode = \"grayscale\"\n",
    "epochs = 20\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Divide the dataset between exemples used for hyperparameters tuning and risk estimation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, test = ff.ready_to_be_used_dataset(\n",
    "    image_size=img_size,\n",
    "    color_mode=color_mode,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the number of batches that must be used for training and validating the model in the hyperparameters tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "true_train_size = len(train) * 4 // 5\n",
    "true_train_size\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Divede the training part between training and validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, valid = train.take(true_train_size), train.skip(true_train_size)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the dimension of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_batch, labels_batch = next(iter(valid))\n",
    "print(\n",
    "    f\"Size of a batch of images {image_batch.shape}\",\n",
    "    f\"Size of a bath of labels {labels_batch.shape}\",\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Baseline model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_list = [\n",
    "    Flatten(input_shape=(img_size, img_size, 1)),\n",
    "    Dense(1000, activation=\"sigmoid\"),\n",
    "    Dense(100, activation=\"sigmoid\"),\n",
    "    Dense(2, activation=\"softmax\"),\n",
    "]\n",
    "\n",
    "model = Sequential(layer_list)\n",
    "model.summary()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_model(\n",
    "    model,\n",
    "    show_shapes=True,\n",
    "    show_dtype=False,\n",
    "    show_layer_names=True,\n",
    "    rankdir=\"TB\",\n",
    "    expand_nested=False,\n",
    "    dpi=96,\n",
    "    layer_range=None,\n",
    "    show_layer_activations=True,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(\n",
    "    optimizer=Adam(),\n",
    "    loss=SparseCategoricalCrossentropy(),\n",
    "    metrics=[\"accuracy\"],\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hyperparameters tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(train, validation_data=valid, epochs=epochs)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ff.performance_plot(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**UNDERFITTING!!** \n",
    "\n",
    "Try with another activation function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_list = [\n",
    "    Flatten(input_shape=(img_size, img_size, 1)),\n",
    "    Dense(1000, activation=\"relu\"),\n",
    "    Dense(100, activation=\"relu\"),\n",
    "    Dense(2, activation=\"softmax\"),\n",
    "]\n",
    "model = Sequential(layer_list)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(\n",
    "    optimizer=Adam(),\n",
    "    loss=SparseCategoricalCrossentropy(),\n",
    "    metrics=[\"accuracy\"],\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(train, validation_data=valid, epochs=epochs)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ff.performance_plot(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems to overfit after the epoch number 11/12.\n",
    "\n",
    "Let's modify the learning rate!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 1e-6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_list = [\n",
    "    Flatten(input_shape=(img_size, img_size, 1)),\n",
    "    Dense(1000, activation=\"relu\"),\n",
    "    Dense(100, activation=\"relu\"),\n",
    "    Dense(2, activation=\"softmax\"),\n",
    "]\n",
    "model = Sequential(layer_list)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(\n",
    "    optimizer=Adam(learning_rate=learning_rate),\n",
    "    loss=SparseCategoricalCrossentropy(),\n",
    "    metrics=[\"accuracy\"],\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(train, validation_data=valid, epochs=epochs)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ff.performance_plot(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a very lower learning rate the neural network reach the optimal performance faster and the learning curve is more smooth.\n",
    "\n",
    "We can deduce that with the default learning rate there were a lot of oscillations, so with a learning rate = 0.000001 we obtain a better exploitation!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Risk estimation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_list = [\n",
    "    Flatten(input_shape=(img_size, img_size, 1)),\n",
    "    Dense(100, activation=\"relu\"),\n",
    "    Dense(2, activation=\"softmax\"),\n",
    "]\n",
    "def_model = Sequential(layer_list)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracies = ff.k_fold_cross_validation(\n",
    "    \"DEF\",\n",
    "    dataset=test,\n",
    "    model_name=layer_list,\n",
    "    epochs=epochs,\n",
    ")\n",
    "\n",
    "\n",
    "avg_loss = 1 - sum(accuracies) / len(accuracies)\n",
    "print(\"Risk estimation (average zero one loss): \", avg_loss)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Many neurons model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try to double the hidden neurons per layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_list = [\n",
    "    Flatten(input_shape=(img_size, img_size, 1)),\n",
    "    Dense(2000, activation=\"relu\"),\n",
    "    Dense(200, activation=\"relu\"),\n",
    "    Dense(2, activation=\"softmax\"),\n",
    "]\n",
    "\n",
    "model = tf.keras.Sequential(layer_list)\n",
    "model.summary()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_model(\n",
    "    model,\n",
    "    show_shapes=True,\n",
    "    show_dtype=False,\n",
    "    show_layer_names=True,\n",
    "    rankdir=\"TB\",\n",
    "    expand_nested=False,\n",
    "    dpi=96,\n",
    "    layer_range=None,\n",
    "    show_layer_activations=True,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(\n",
    "    optimizer=Adam(learning_rate=learning_rate),\n",
    "    loss=SparseCategoricalCrossentropy(),\n",
    "    metrics=[\"accuracy\"],\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hyperparameters tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(train, validation_data=valid, epochs=epochs)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ff.performance_plot(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Risk estimation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Many layers model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_list = [\n",
    "    Flatten(input_shape=(img_size, img_size, 1)),\n",
    "    Dense(750, activation=\"relu\"),\n",
    "    Dense(250, activation=\"relu\"),\n",
    "    Dense(75, activation=\"relu\"),\n",
    "    Dense(25, activation=\"relu\"),\n",
    "    Dense(2, activation=\"softmax\"),\n",
    "]\n",
    "model = Sequential(layer_list)\n",
    "model.summary()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_model(\n",
    "    model,\n",
    "    show_shapes=True,\n",
    "    show_dtype=False,\n",
    "    show_layer_names=True,\n",
    "    rankdir=\"TB\",\n",
    "    expand_nested=False,\n",
    "    dpi=96,\n",
    "    layer_range=None,\n",
    "    show_layer_activations=True,\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(\n",
    "    optimizer=Adam(learning_rate=learning_rate),\n",
    "    loss=SparseCategoricalCrossentropy(),\n",
    "    metrics=[\"accuracy\"],\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hyperparameter tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(train, validation_data=valid, epochs=epochs)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ff.performance_plot(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Risk estimation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracies = ff.k_fold_cross_validation(\n",
    "    \"MANY_L\",\n",
    "    test,\n",
    "    layer_list,\n",
    "    learning_rate=learning_rate,\n",
    "    epochs=epochs,\n",
    ")\n",
    "\n",
    "avg_loss = 1 - sum(accuracies) / len(accuracies)\n",
    "print(\"Risk estimation (average zero one loss): \", avg_loss)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.2 ('venv': venv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "0c89f70f669f0ea3789c57a403510e9f71ee3829d9d70c9b2bdd58feb836f7dd"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}