papers/identity_link_unnormalized_numeric_feature.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Identity Links Deep Neural Networks\n",
    "\n",
    "Shallow Deep neural networks are the 'defacto' model for structured data consisting only of numeric data (e.g., sensor measurements).\n",
    "\n",
    "When numeric data columns are not normalized within the same range, we find that many times a deep neural network will either not converge, or overfit to the training data. We will demonstrate that a shallow deep neural network can be trained to converge without overfitting on non-normalized numeric data by adding an identity link from the input vector to each shallow layer.\n",
    "\n",
    "We observe from this technique the following:\n",
    "\n",
    "    1. Introduces regularization into the model (preventing overfitting).\n",
    "    2. Adds stability to the slope of the valuation loss/accuracy with non-normalized data.\n",
    "    \n",
    "While the identity link adds some parameters, we add it as a concatenation vector operation. This is a fast operation, and only adds a nominal number of parameters at each layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Datasets\n",
    "\n",
    "In this notebook, we use two well-known numeric only datasets: 'iris' and 'wine' for demonstration. Both datasets consists of numeric data only with three output classes. \n",
    "\n",
    "### Iris Dataset\n",
    "\n",
    "This dataset consists of 150 examples for three output classes (50 each). The data consists of 4 numeric columns.\n",
    "\n",
    "See (UCI Machine Learning Repository for more details)[https://archive.ics.uci.edu/ml/datasets/iris]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "iris = datasets.load_iris()\n",
    "X_iris = iris.data\n",
    "Y_iris = iris.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wine Dataset\n",
    "\n",
    "This dataset consists of 178 examples for three output classes (~60 each). The data consists of 13 numeric columns.\n",
    "\n",
    "See (UCI Machine Learning Repository for more details)[https://archive.ics.uci.edu/ml/datasets/wine]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wine = datasets.load_wine()\n",
    "X_wine = wine.data\n",
    "Y_wine = wine.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models\n",
    "\n",
    "For both datasets, we will use the same shallow deep model, with the only difference being the input vector:\n",
    "\n",
    "        input layer : 10 node dense layer\n",
    "        hidden layer: 10 node dense layer\n",
    "        output layer: 3 node dense layer\n",
    "        \n",
    "We will train using non-normalized data on two versions of the model for each dataset. In the first version, we will train w/o the identity linkand then with the identity link."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Iris Model\n",
    "\n",
    "    iris_model  : model w/o identity link\n",
    "    iris_modelx : model with identity link"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import Input, Model\n",
    "from tensorflow.keras.layers import Dense, Concatenate\n",
    "\n",
    "def make_iris():\n",
    "    inputs = Input(shape=(4,))\n",
    "    x = Dense(10, activation='relu')(inputs)\n",
    "    x = Dense(10, activation='relu')(x)\n",
    "    outputs = Dense(3, activation='softmax')(x)\n",
    "    return Model(inputs, outputs)\n",
    "\n",
    "iris_model = make_iris()\n",
    "iris_model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])\n",
    "iris_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_iris_x():\n",
    "    inputs = Input(shape=(4,))\n",
    "    x = Dense(10, activation='relu')(inputs)\n",
    "    x = Concatenate()([inputs, x])\n",
    "    x = Dense(10, activation='relu')(x)\n",
    "    x = Concatenate()([inputs, x])\n",
    "    outputs = Dense(3, activation='softmax')(x)\n",
    "    return Model(inputs, outputs)\n",
    "\n",
    "iris_modelx = make_iris_x()\n",
    "iris_modelx.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])\n",
    "iris_modelx.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wine Model\n",
    "\n",
    "    wine_model  : model w/o identity link\n",
    "    wine_modelx : model with identity link"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_wine():\n",
    "    inputs = Input(shape=(13,))\n",
    "    x = Dense(10, activation='relu')(inputs)\n",
    "    x = Dense(10, activation='relu')(x)\n",
    "    outputs = Dense(3, activation='softmax')(x)\n",
    "    return Model(inputs, outputs)\n",
    "\n",
    "wine_model = make_wine()\n",
    "wine_model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])\n",
    "wine_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_wine_x():\n",
    "    inputs = Input(shape=(13,))\n",
    "    x = Dense(10, activation='relu')(inputs)\n",
    "    x = Concatenate()([inputs, x])\n",
    "    x = Dense(10, activation='relu')(x)\n",
    "    x = Concatenate()([inputs, x])\n",
    "    outputs = Dense(3, activation='softmax')(x)\n",
    "    return Model(inputs, outputs)\n",
    "\n",
    "wine_modelx = Model(inputs, outputs)\n",
    "wine_modelx.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])\n",
    "wine_modelx.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "\n",
    "Each traing session will train for 30 epochs for stocastic gradient descent (batch=1).\n",
    "\n",
    "### Iris w/o identity Link"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " for _ in range(3):\n",
    "        print(\"ITER\", _)\n",
    "        model = make_iris()\n",
    "        model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])\n",
    "        model.fit(X_iris, Y_iris, epochs=30, batch_size=1, verbose=1, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Results (Non-Identity)\n",
    "\n",
    "    ZERO - Number of epochs where valuation accuracy < 3%\n",
    "    AVE  - Average valuation accuracy across 30 epochs.\n",
    "\n",
    "    ITER 1: ZERO 9, AVE 30%\n",
    "    ITER 2: ZERO 16, AVE 25%\n",
    "    ITER 3: ZERO 1, AVE 62%\n",
    "\n",
    "    Total number of ZERO: 26\n",
    "    Average Acc across iterations: 39"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " for _ in range(3):\n",
    "        print(\"ITER\", _)\n",
    "        model = make_iris_x()\n",
    "        model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])\n",
    "        model.fit(X_iris, Y_iris, epochs=30, batch_size=1, verbose=1, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Results (Identity)\n",
    "\n",
    "    ZERO - Number of epochs where valuation accuracy < 3%\n",
    "    AVE  - Average valuation accuracy across 30 epochs.\n",
    "\n",
    "    ITER 1: ZERO 7, AVE 30%\n",
    "    ITER 2: ZERO 3, AVE 25%\n",
    "    ITER 3: ZERO 5, AVE 62%\n",
    "\n",
    "    Total number of ZERO: 15\n",
    "    Average Acc across iterations: 49"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wine w/o Identity Link\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for _ in range(3):\n",
    "        print(\"ITER\", _)\n",
    "        model = make_wine()\n",
    "        model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])\n",
    "        model.fit(X_wine, Y_wine, epochs=30, batch_size=1, verbose=1, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Results (Non-Identity)\n",
    "\n",
    "    ZERO - Number of epochs where valuation accuracy < 3%\n",
    "    AVE  - Average valuation accuracy across 30 epochs.\n",
    "\n",
    "    ITER 1: ZERO 28, AVE 3%\n",
    "    ITER 2: ZERO 14, AVE 28%\n",
    "    ITER 3: ZERO 14, AVE 24%\n",
    "\n",
    "    Total number of ZERO: 56\n",
    "    Average Acc across iterations: 18%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for _ in range(3):\n",
    "        print(\"ITER\", _)\n",
    "        model = make_wine_x()\n",
    "        model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])\n",
    "        model.fit(X_wine, Y_wine, epochs=30, batch_size=1, verbose=1, validation_split=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Results (Non-Identity)\n",
    "\n",
    "    ZERO - Number of epochs where valuation accuracy < 3%\n",
    "    AVE  - Average valuation accuracy across 30 epochs.\n",
    "\n",
    "    ITER 1: ZERO 5, AVE 58%\n",
    "    ITER 2: ZERO 1, AVE 81%\n",
    "    ITER 3: ZERO 2, AVE 81%\n",
    "\n",
    "    Total number of ZERO: 56\n",
    "    Average Acc across iterations: 18%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}