examples/Recover Best Models from Experiment Log.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=https://raw.githubusercontent.com/autonomio/hyperio/master/logo.png>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to recover best model from experiment log?\n",
    "Due to system error or other reason where scan_object is no longer available, it's still possible to get best model/s using nothing but the experiment log. In the below notebook you will learn exactly how.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import talos\n",
    "import wrangle\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we'll have to perform the `Scan()` experiment to produce the experiment log. Because the experiment log is stored on local machine, interrupted `Scan()` or other reason will not affect its availability. The experiment log is updated after each permutation; it contains an up-to-date record of the experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the data\n",
    "x, y = talos.templates.datasets.iris()\n",
    "x_train, y_train, x_val, y_val = wrangle.array_split(x, y, .3)\n",
    "\n",
    "# set the parameter space boundary\n",
    "p = {'activation':['relu', 'elu'],\n",
    "     'optimizer': ['Nadam', 'Adam'],\n",
    "     'losses': ['logcosh'],\n",
    "     'shapes': ['brick'],\n",
    "     'first_neuron': [16, 32, 64, 128],\n",
    "     'hidden_layers':[0, 1, 2, 3],\n",
    "     'dropout': [.2, .3, .4],\n",
    "     'batch_size': [20, 30, 40, 50],\n",
    "     'epochs': [10]}\n",
    "\n",
    "# define the input model\n",
    "def iris_model(x_train, y_train, x_val, y_val, params):\n",
    "\n",
    "    model = Sequential()\n",
    "    model.add(Dense(params['first_neuron'], input_dim=4, activation=params['activation']))\n",
    "\n",
    "    talos.utils.hidden_layers(model, params, 3)\n",
    "\n",
    "    model.add(Dense(3, activation='softmax'))\n",
    "    model.compile(optimizer=params['optimizer'], loss=params['losses'], metrics=['acc'])\n",
    "\n",
    "    out = model.fit(x_train, y_train, callbacks=[talos.utils.ExperimentLogCallback('minimal_iris', params)],\n",
    "                     batch_size=params['batch_size'],\n",
    "                     epochs=params['epochs'],\n",
    "                     validation_data=[x_val, y_val],\n",
    "                     verbose=0)\n",
    "\n",
    "    return out, model\n",
    "\n",
    "# start the experiment\n",
    "scan_object = talos.Scan(x=x_train,\n",
    "                         y=y_train,\n",
    "                         x_val=x_val,\n",
    "                         y_val=y_val,\n",
    "                         model=iris_model,\n",
    "                         experiment_name='minimal_iris',\n",
    "                         params=p,\n",
    "                         round_limit=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can assume the case where we no longer have access to the `scan_object`. In this `Scan(...experiment_name...)` was set to \"reactivate\" so we'll find a folder with that name in the present working directory. Next we have to find out what is the name of the experiment log."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the name of the experiment log\n",
    "!ls -lhtr minimal_iris"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What you want to do, is get the name of the `.csv` file you want to use, and use it as part of the input for `experiment_log` in the next step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from talos.utils.recover_best_model import recover_best_model\n",
    "\n",
    "results, models = recover_best_model(x_train=x_train,\n",
    "                                     y_train=y_train,\n",
    "                                     x_val=x_val,\n",
    "                                     y_val=y_val,\n",
    "                                     experiment_log='minimal_iris/012620102735.csv',\n",
    "                                     input_model=iris_model,\n",
    "                                     n_models=5,\n",
    "                                     task='multi_label')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can access the cross-validation results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also access the models and make predictions with them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "models[0].predict(x_val)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}