examples/Recover Best Models from Experiment Log.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=https://raw.githubusercontent.com/autonomio/hyperio/master/logo.png>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to recover best model from experiment log?\n",
    "Due to system error or other reason where scan_object is no longer available, it's still possible to get best model/s using nothing but the experiment log. In the below notebook you will learn exactly how.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.insert(0, '/Users/mikko/Documents/GitHub/talos/')\n",
    "\n",
    "import sys\n",
    "sys.path.insert(0, '/Users/mikko/Documents/GitHub/wrangle/')\n",
    "\n",
    "import talos\n",
    "import wrangle\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we'll have to perform the `Scan()` experiment to produce the experiment log. Because the experiment log is stored on local machine, interrupted `Scan()` or other reason will not affect its availability. The experiment log is updated after each permutation; it contains an up-to-date record of the experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10/10 [00:34<00:00,  3.28s/it]\n"
     ]
    }
   ],
   "source": [
    "# load the data\n",
    "x, y = talos.templates.datasets.iris()\n",
    "x_train, y_train, x_val, y_val = wrangle.array_split(x, y, .3)\n",
    "\n",
    "# set the parameter space boundary\n",
    "p = {'activation':['relu', 'elu'],\n",
    "     'optimizer': ['Nadam', 'Adam'],\n",
    "     'losses': ['logcosh'],\n",
    "     'shapes': ['brick'],\n",
    "     'first_neuron': [16, 32, 64, 128],\n",
    "     'hidden_layers':[0, 1, 2, 3],\n",
    "     'dropout': [.2, .3, .4],\n",
    "     'batch_size': [20, 30, 40, 50],\n",
    "     'epochs': [10]}\n",
    "\n",
    "# define the input model\n",
    "def iris_model(x_train, y_train, x_val, y_val, params):\n",
    "\n",
    "    model = Sequential()\n",
    "    model.add(Dense(params['first_neuron'], input_dim=4, activation=params['activation']))\n",
    "\n",
    "    talos.utils.hidden_layers(model, params, 3)\n",
    "\n",
    "    model.add(Dense(3, activation='softmax'))\n",
    "    model.compile(optimizer=params['optimizer'], loss=params['losses'], metrics=['acc'])\n",
    "\n",
    "    out = model.fit(x_train, y_train, callbacks=[talos.utils.ExperimentLogCallback('minimal_iris', params)],\n",
    "                     batch_size=params['batch_size'],\n",
    "                     epochs=params['epochs'],\n",
    "                     validation_data=[x_val, y_val],\n",
    "                     verbose=0)\n",
    "\n",
    "    return out, model\n",
    "\n",
    "# start the experiment\n",
    "scan_object = talos.Scan(x=x_train,\n",
    "                         y=y_train,\n",
    "                         x_val=x_val,\n",
    "                         y_val=y_val,\n",
    "                         model=iris_model,\n",
    "                         experiment_name='reactivate',\n",
    "                         params=p,\n",
    "                         round_limit=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can assume the case where we no longer have access to the `scan_object`. In this `Scan(...experiment_name...)` was set to \"reactivate\" so we'll find a folder with that name in the present working directory. Next we have to find out what is the name of the experiment log."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 96\r\n",
      "-rw-r--r--  1 mikko  staff    16K Sep 26 20:17 092619200831.csv\r\n",
      "-rw-r--r--  1 mikko  staff   1.3K Sep 26 20:18 092619201824.csv\r\n",
      "-rw-r--r--  1 mikko  staff   1.3K Sep 26 22:13 092619221236.csv\r\n",
      "-rw-r--r--  1 mikko  staff   1.3K Sep 26 22:18 092619221803.csv\r\n",
      "-rw-r--r--  1 mikko  staff   1.3K Sep 26 22:31 092619223042.csv\r\n",
      "-rw-r--r--  1 mikko  staff   1.3K Sep 26 22:35 092619223459.csv\r\n",
      "-rw-r--r--  1 mikko  staff   1.3K Sep 26 22:35 092619223524.csv\r\n",
      "-rw-r--r--  1 mikko  staff   1.3K Sep 26 22:56 092619225556.csv\r\n",
      "-rw-r--r--  1 mikko  staff   1.3K Sep 26 23:04 092619230425.csv\r\n"
     ]
    }
   ],
   "source": [
    "# get the name of the experiment log\n",
    "!ls -lhtr reactivate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case it will be the most recent one `092619223042.csv` so let's go ahead and recover the best models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from talos.utils.recover_best_model import recover_best_model\n",
    "results, models = recover_best_model(x_train=x_train,\n",
    "                                     y_train=y_train,\n",
    "                                     x_val=x_val,\n",
    "                                     y_val=y_val,\n",
    "                                     experiment_log='reactivate/092619221803.csv',\n",
    "                                     input_model=iris_model,\n",
    "                                     n_models=5,\n",
    "                                     task='multi_label')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can access the cross-validation results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>round_epochs</th>\n",
       "      <th>val_loss</th>\n",
       "      <th>val_acc</th>\n",
       "      <th>loss</th>\n",
       "      <th>acc</th>\n",
       "      <th>activation</th>\n",
       "      <th>batch_size</th>\n",
       "      <th>dropout</th>\n",
       "      <th>epochs</th>\n",
       "      <th>first_neuron</th>\n",
       "      <th>hidden_layers</th>\n",
       "      <th>losses</th>\n",
       "      <th>optimizer</th>\n",
       "      <th>shapes</th>\n",
       "      <th>crossval_mean_f1score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10</td>\n",
       "      <td>0.028760</td>\n",
       "      <td>0.955556</td>\n",
       "      <td>0.031489</td>\n",
       "      <td>0.885714</td>\n",
       "      <td>relu</td>\n",
       "      <td>20</td>\n",
       "      <td>0.3</td>\n",
       "      <td>10</td>\n",
       "      <td>128</td>\n",
       "      <td>2</td>\n",
       "      <td>logcosh</td>\n",
       "      <td>Adam</td>\n",
       "      <td>brick</td>\n",
       "      <td>0.930236</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>10</td>\n",
       "      <td>0.028902</td>\n",
       "      <td>0.933333</td>\n",
       "      <td>0.029437</td>\n",
       "      <td>0.866667</td>\n",
       "      <td>elu</td>\n",
       "      <td>50</td>\n",
       "      <td>0.3</td>\n",
       "      <td>10</td>\n",
       "      <td>128</td>\n",
       "      <td>2</td>\n",
       "      <td>logcosh</td>\n",
       "      <td>Adam</td>\n",
       "      <td>brick</td>\n",
       "      <td>0.900427</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>10</td>\n",
       "      <td>0.023247</td>\n",
       "      <td>0.888889</td>\n",
       "      <td>0.020156</td>\n",
       "      <td>0.942857</td>\n",
       "      <td>elu</td>\n",
       "      <td>30</td>\n",
       "      <td>0.3</td>\n",
       "      <td>10</td>\n",
       "      <td>128</td>\n",
       "      <td>2</td>\n",
       "      <td>logcosh</td>\n",
       "      <td>Adam</td>\n",
       "      <td>brick</td>\n",
       "      <td>0.980606</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10</td>\n",
       "      <td>0.044717</td>\n",
       "      <td>0.866667</td>\n",
       "      <td>0.048760</td>\n",
       "      <td>0.780952</td>\n",
       "      <td>relu</td>\n",
       "      <td>30</td>\n",
       "      <td>0.4</td>\n",
       "      <td>10</td>\n",
       "      <td>128</td>\n",
       "      <td>2</td>\n",
       "      <td>logcosh</td>\n",
       "      <td>Adam</td>\n",
       "      <td>brick</td>\n",
       "      <td>0.529231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>10</td>\n",
       "      <td>0.047718</td>\n",
       "      <td>0.866667</td>\n",
       "      <td>0.052877</td>\n",
       "      <td>0.761905</td>\n",
       "      <td>elu</td>\n",
       "      <td>30</td>\n",
       "      <td>0.3</td>\n",
       "      <td>10</td>\n",
       "      <td>64</td>\n",
       "      <td>1</td>\n",
       "      <td>logcosh</td>\n",
       "      <td>Adam</td>\n",
       "      <td>brick</td>\n",
       "      <td>0.860539</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   round_epochs  val_loss   val_acc      loss       acc activation  \\\n",
       "0            10  0.028760  0.955556  0.031489  0.885714       relu   \n",
       "9            10  0.028902  0.933333  0.029437  0.866667        elu   \n",
       "2            10  0.023247  0.888889  0.020156  0.942857        elu   \n",
       "4            10  0.044717  0.866667  0.048760  0.780952       relu   \n",
       "8            10  0.047718  0.866667  0.052877  0.761905        elu   \n",
       "\n",
       "   batch_size  dropout  epochs  first_neuron  hidden_layers   losses  \\\n",
       "0          20      0.3      10           128              2  logcosh   \n",
       "9          50      0.3      10           128              2  logcosh   \n",
       "2          30      0.3      10           128              2  logcosh   \n",
       "4          30      0.4      10           128              2  logcosh   \n",
       "8          30      0.3      10            64              1  logcosh   \n",
       "\n",
       "  optimizer shapes  crossval_mean_f1score  \n",
       "0      Adam  brick               0.930236  \n",
       "9      Adam  brick               0.900427  \n",
       "2      Adam  brick               0.980606  \n",
       "4      Adam  brick               0.529231  \n",
       "8      Adam  brick               0.860539  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also access the models and make predictions with them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[4.0995691e-02, 7.6433158e-01, 1.9467270e-01],\n",
       "       [5.5384491e-02, 8.0346000e-01, 1.4115542e-01],\n",
       "       [1.4972138e-01, 7.5501865e-01, 9.5259979e-02],\n",
       "       [1.1326348e-01, 7.1604651e-01, 1.7069000e-01],\n",
       "       [3.7939314e-02, 6.4578909e-01, 3.1627160e-01],\n",
       "       [7.1953669e-02, 7.3772454e-01, 1.9032180e-01],\n",
       "       [1.9050457e-02, 5.2020442e-01, 4.6074522e-01],\n",
       "       [3.7893724e-02, 6.0179770e-01, 3.6030859e-01],\n",
       "       [9.7751850e-03, 3.3405378e-01, 6.5617102e-01],\n",
       "       [1.1282027e-02, 3.9736888e-01, 5.9134912e-01],\n",
       "       [3.4884610e-03, 2.8260693e-01, 7.1390456e-01],\n",
       "       [1.4066804e-02, 4.4939899e-01, 5.3653419e-01],\n",
       "       [9.8388308e-01, 1.5294192e-02, 8.2262064e-04],\n",
       "       [9.4405776e-03, 2.6316279e-01, 7.2739667e-01],\n",
       "       [5.3356937e-03, 2.5790238e-01, 7.3676193e-01],\n",
       "       [6.6629532e-03, 2.9248431e-01, 7.0085275e-01],\n",
       "       [1.0408297e-02, 2.8865287e-01, 7.0093888e-01],\n",
       "       [9.2165405e-03, 4.4728053e-01, 5.4350299e-01],\n",
       "       [6.1816044e-02, 7.1130496e-01, 2.2687899e-01],\n",
       "       [5.8197163e-02, 7.0744330e-01, 2.3435953e-01],\n",
       "       [9.7906607e-01, 1.9747239e-02, 1.1866244e-03],\n",
       "       [4.9119215e-02, 6.6068804e-01, 2.9019269e-01],\n",
       "       [9.7612959e-01, 2.2444952e-02, 1.4254292e-03],\n",
       "       [4.5029860e-02, 6.6044801e-01, 2.9452211e-01],\n",
       "       [9.9142039e-01, 8.2095861e-03, 3.7001175e-04],\n",
       "       [9.7515500e-01, 2.3289582e-02, 1.5554430e-03],\n",
       "       [5.3465478e-02, 6.7216325e-01, 2.7437130e-01],\n",
       "       [3.2891510e-03, 1.4500402e-01, 8.5170686e-01],\n",
       "       [4.0943369e-02, 7.4083865e-01, 2.1821795e-01],\n",
       "       [9.7216946e-01, 2.5830602e-02, 1.9999193e-03],\n",
       "       [5.3125862e-02, 7.0550537e-01, 2.4136871e-01],\n",
       "       [1.1228154e-01, 7.2705197e-01, 1.6066651e-01],\n",
       "       [2.1662652e-02, 5.3180271e-01, 4.4653463e-01],\n",
       "       [4.6057135e-02, 7.8316230e-01, 1.7078057e-01],\n",
       "       [3.8668580e-02, 5.8559459e-01, 3.7573683e-01],\n",
       "       [2.0084916e-02, 4.8307988e-01, 4.9683511e-01],\n",
       "       [2.2133207e-02, 5.7616937e-01, 4.0169743e-01],\n",
       "       [9.6933258e-01, 2.8047977e-02, 2.6194439e-03],\n",
       "       [6.2396564e-02, 6.9619435e-01, 2.4140903e-01],\n",
       "       [9.8845774e-01, 1.0924198e-02, 6.1809190e-04],\n",
       "       [1.1867868e-02, 2.8613701e-01, 7.0199513e-01],\n",
       "       [1.3917084e-02, 4.5086253e-01, 5.3522038e-01],\n",
       "       [3.7958041e-02, 7.3275220e-01, 2.2928977e-01],\n",
       "       [1.5825152e-02, 5.4144788e-01, 4.4272691e-01],\n",
       "       [9.7893941e-01, 1.9864958e-02, 1.1956602e-03]], dtype=float32)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "models[0].predict(x_val)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}