examples/A Very Short Introduction to Hyperparameter Optimization of Keras Models with Talos.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://raw.githubusercontent.com/autonomio/hyperio/master/logo.png' width=250px>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting Diabetes with Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Very Short Introduction to Hyperparameter Optimization with Talos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of a Talos experiment, is to find a set of suitable hyperparameters for Keras model. In order to do this, you need to have three things: \n",
    "\n",
    "- Keras model \n",
    "- Talos hyperparameter dictionary\n",
    "- Talos experiment configuration\n",
    "\n",
    "Below we will briefly overview each."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Keras Model\n",
    "\n",
    "As a model, any Keras model will do. Let's consider as an example a very simple model that makes a prediction on the classic *Pima Indians Diabetes* dataset. A brief overview of the dataset can be found [here](https://www.kaggle.com/uciml/pima-indians-diabetes-database) and the dataset we will use can be found [here](https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv). The below model does not require you to separately download the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from numpy import loadtxt\n",
    "\n",
    "dataset = loadtxt(\"https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv\", delimiter=\",\")\n",
    "x = dataset[:,0:8]\n",
    "y = dataset[:,8]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "\n",
    "def diabetes():\n",
    "    \n",
    "    model = Sequential()\n",
    "    model.add(Dense(12, input_dim=8, activation='relu'))\n",
    "    model.add(Dense(8, activation='relu'))\n",
    "    model.add(Dense(1, activation='sigmoid'))\n",
    "    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "    model.fit(X, Y, epochs=100, batch_size=10, verbose=0)\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Talos Hyperparameter Dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's prepare for an experiment where we will optimize against three common attributes:\n",
    "\n",
    "- neurons on the first layer\n",
    "- activations\n",
    "- batch_sizes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.activations import relu, elu\n",
    "\n",
    "p = {\n",
    "    'first_neuron': [12, 24, 48],\n",
    "    'activation': [relu, elu],\n",
    "    'batch_size': [10, 20, 30]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configuring the Keras Model for Talos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to prepare a Keras model for a Talos experiment, we need to do four things:\n",
    "\n",
    "- add input parameters to the function\n",
    "- replace the hyperparameter inputs with references to params dictionary\n",
    "- make sure model.fit() stores the history object\n",
    "- modify the output of the model\n",
    "\n",
    "These steps are always the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add input parameters to the function\n",
    "def diabetes(x_train, y_train, x_val, y_val, params):\n",
    "    \n",
    "    # replace the hyperparameter inputs with references to params dictionary \n",
    "    model = Sequential()\n",
    "    model.add(Dense(params['first_neuron'], input_dim=8, activation=params['activation']))\n",
    "    #model.add(Dense(8, activation=params['activation']))\n",
    "    model.add(Dense(1, activation='sigmoid'))\n",
    "    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "    \n",
    "    # make sure history object is returned by model.fit()\n",
    "    out = model.fit(x, y,\n",
    "                    epochs=100,\n",
    "                    batch_size=params['batch_size'],\n",
    "                    #validation_split=.3,\n",
    "                    verbose=0)\n",
    "    \n",
    "    # modify the output model\n",
    "    return out, model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it, there is nothing more to it. A more complicated experiment would just entail more of the same in terms of the way the params dictionary references are made. Otherwise the changes would always be exactly the same."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Talos Experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Talos experiment is performed through the Scan() command. In case you don't have Talos installed already, you can do that now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, '/Users/mikko/Documents/GitHub/talos/')\n",
    "import talos as ta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While many configurations are possible, the only things that you absolutely must input to a Talos experiment are: \n",
    "\n",
    "- x\n",
    "- y\n",
    "- params (the dictionary 'p' we created above)\n",
    "- model (the 'diabetes' we created above)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t = ta.Scan(x, y, p, diabetes)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}