examples/00_interfacing_DEHB.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How to read this notebook\n",
    "\n",
    "This notebook is designed to serve as a high-level, highly abstracted view of DEHB and how it can be used. The examples here are mere placeholders and *only* offer an interface to run DEHB on toy or actual problems.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import warnings\n",
    "import numpy as np\n",
    "import ConfigSpace\n",
    "from typing import Dict, Union, List\n",
    "\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting started with DEHB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DEHB was designed to be an algorithm for Hyper Parameter Optimization (HPO). DEHB uses Differential Evolution (DE) under-the-hood as an Evolutionary Algorithm to power the black-box optimization that HPO problems pose. DE is a black-box optimization algorithm that generates candidate configurations $x$, to the black-box function $f(x)$, that is being optimized. The $x$ is evaluated by the black-box and the corresponding response $y$ is made available to the DE algorithm, which can then use this observation ($x$, $y$), along with previous such observations, to suggest a new candidate $x$ for the next evaluation. \n",
    "\n",
    "DEHB also uses Hyperband along with DE, to allow for cheaper approximations of the actual evaluations of $x$. Let $f(x)$ be the validation error of training a multilayer perceptron (MLP) on the complete training set. Multi-fidelity algorithms such as Hyperband, allow for cheaper approximations along a possible *fidelity*. For the MLP, a subset of the dataset maybe a cheaper approximation to the full data set evaluation. Whereas the fidelity can be quantifies as the fraction of the dataset used to evaluate the configuration $x$, instead of the full dataset. Such approximations can allow sneak-peek into the black-box, potentially revealing certain landscape feature of *f(x)*, thus rendering it a *gray*-box and not completely opaque and black! \n",
    "\n",
    "The $z$ parameter is the fidelity parameter to the black-box function. If $z \\in [budget_{min}, budget_{max}]$, then $f(x, budget_{max})$ would be equivalent to the black-box case of $f(x)$.\n",
    "\n",
    "![boxes](imgs/black-gray-box.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HPO algorithms optimize such black/gray box by wrapping around this *target* function an interface, by which the algorithms can suggest new $x$ and also consume the result of the corresponding evaluation to store a collection of such ($x$, $y$) pairs. Therefore, to run DEHB, the most essential component required as input is the target function to optimize. Since DEHB can leverage a Hyperband, the target function interface should account for possible input of fidelity too. \n",
    "\n",
    "### Sample interface for target function that DEHB optimizes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def target_function(\n",
    "    x: Union[ConfigSpace.Configuration, List, np.array], \n",
    "    budget: Union[int, float] = None,\n",
    "    **kwargs\n",
    ") -> Dict:\n",
    "    \"\"\" Target/objective function to optimize\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    x : configuration that DEHB wants to evaluate\n",
    "    budget : parameter determining cheaper evaluations\n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    dict\n",
    "    \"\"\"\n",
    "    # ...\n",
    "    # write your code here\n",
    "    # ...\n",
    "    \n",
    "    # remove the code snippet below\n",
    "    start = time.time()\n",
    "    y = np.random.uniform()  # placeholder response of evaluation\n",
    "    time.sleep(budget)       # simulates runtime (mostly proportional to fidelity)\n",
    "    cost = time.time() - start\n",
    "    \n",
    "    # result dict passed to DE/DEHB as function evaluation output\n",
    "    result = {\n",
    "        \"fitness\": y,  # must-have key that DE/DEHB minimizes\n",
    "        \"cost\": cost,  # must-have key that associates cost/runtime \n",
    "        \"info\": dict() # optional key containing a dictionary of additional info\n",
    "    }\n",
    "    return result    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This `target_function` is the problem that needs to be solved, or the function to be optimized. The other prerequisite for this function is therefore the domain for its input $x$. In other words, the definition and constraints of the *search space* for DEHB. \n",
    "\n",
    "The DE component inside DEHB, **assumes that the input domain is scaled to a unit hypercube**. This is essential for effective search. If the [ConfigSpace](https://pypi.org/project/ConfigSpace/) library is used to define the domain of $x$, or the parameters of the search space, DEHB can internally handle the scaling to and from the unit hypercube required for search. If ConfigSpace is not used, one needs to additionally handle the scaling of the parameters as an extra interface between DEHB and the target function (or encode it within the target function). \n",
    "\n",
    "For this template notebook, we will illustrate how a ConfigSpace parameter space can be created."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining a sample search space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Configuration space object:\n",
      "  Hyperparameters:\n",
      "    x0, Type: UniformFloat, Range: [3.0, 10.0], Default: 6.5\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import ConfigSpace\n",
    "\n",
    "\n",
    "def create_search_space():\n",
    "    # Creating a one-dimensional search space of real numbers in [3, 10]\n",
    "    cs = ConfigSpace.ConfigurationSpace()\n",
    "    cs.add_hyperparameter(ConfigSpace.UniformFloatHyperparameter(\"x0\", lower=3, upper=10, log=False))\n",
    "    return cs\n",
    "\n",
    "\n",
    "cs = create_search_space()\n",
    "print(cs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "# Finding dimensionality of search space\n",
    "dimensions = len(cs.get_hyperparameters())\n",
    "print(dimensions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Configuration:\n",
       "  x0, Value: 3.716302229868112"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Sampling a random configuration\n",
    "cs.sample_configuration()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [ConfigSpace documentation](https://automl.github.io/ConfigSpace/master/index.html) can be referred to for more complicated search space creation.\n",
    "\n",
    "In a similar vein, for a complete gray-box definition, the fidelity domain needs to be defined too. For the earlier example of dataset fractions, the fidelity upper limit cannot clearly exceed 1, and therefore $[0.3, 1]$ is a legitimate definition for such a fidelity. In this template example, we shall simply define the lower and upper range of the fidelity as two parameters that can be input to DEHB. Given that fidelity is being used to simulate cost of runtime in our sample `target_function`, we shall use a reasonable time range as a placeholder for the fidelity in this case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining fidelity/budget range for the target function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_budget, max_budget = (0.1, 3) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above definitions are all the information that DEHB needs about a problem. We are now in a position to call upon DEHB and start running it, to tune $x$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Instantiating and running DEHB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(\"../\")\n",
    "from dehb import DEHB\n",
    "\n",
    "\n",
    "dehb = DEHB(\n",
    "    f=target_function,\n",
    "    dimensions=dimensions,\n",
    "    cs=cs,\n",
    "    min_budget=min_budget,\n",
    "    max_budget=max_budget,\n",
    "    output_path=\"./temp\",\n",
    "    n_workers=1        # set to >1 to utilize parallel workers\n",
    ")\n",
    "\n",
    "# NOTE: the other hyperparameters to DEHB have been set to certain defaults that were \n",
    "# empirically determined through related literature, ablation analysis and other experiments,\n",
    "# but can be tuned as desired"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "?dehb.run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DEHB allows the option of 3 different resources for its runtime budget:\n",
    "#### 1) Running DEHB for a certain number of (successive halving) *brackets*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Configuration:\n",
      "  x0, Value: 4.060258498267547\n",
      "\n"
     ]
    }
   ],
   "source": [
    "_, _, _ = dehb.run(brackets=1, verbose=False, save_intermediate=True)\n",
    "print(dehb.vector_to_configspace(dehb.inc_config))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2) Running DEHB for total number of *function evaluations*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-10-22 14:45:56.117 | INFO     | dehb.optimizers.dehb:reset:107 - \n",
      "\n",
      "RESET at 10/22/21 14:45:56 CEST\n",
      "\n",
      "\n",
      "(Configuration:\n",
      "  x0, Value: 3.724555206841792\n",
      ", 0.0938589687572785)\n"
     ]
    }
   ],
   "source": [
    "# allows optimization to restart from the beginning by forgetting al observations\n",
    "dehb.reset()  \n",
    "\n",
    "_, _, _ = dehb.run(fevals=20, verbose=False, save_intermediate=True)\n",
    "print(dehb.get_incumbents())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3) Running DEHB for total amount of *wallclock time*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-10-22 14:45:58.567 | INFO     | dehb.optimizers.dehb:reset:107 - \n",
      "\n",
      "RESET at 10/22/21 14:45:58 CEST\n",
      "\n",
      "\n",
      "(Configuration:\n",
      "  x0, Value: 4.341818535733585\n",
      ", 3.653636256717441e-05)\n"
     ]
    }
   ],
   "source": [
    "# allows optimization to restart from the beginning by forgetting all observations\n",
    "dehb.reset()  \n",
    "\n",
    "_, _, _ = dehb.run(total_cost=10, verbose=False, save_intermediate=True)  # run for 10s\n",
    "print(dehb.get_incumbents())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each `dehb` object initialized maintains a `log` file in the `output_path` specified, where the progress and other debugging information is updated. While every alternative DEHB evaluation (and after full optimization), an `incumbent.json` file is written to disk `output_path`, with the incumbent (best seen so far) configuration and its corresponding score. \n",
    "\n",
    "\n",
    "We now rerun DEHB in parallel with 2 workers, and show that the incumbents can be retrieved in any of the following manner:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(Configuration:\n",
      "  x0, Value: 4.610766436763522\n",
      ", 0.007774399252232556)\n"
     ]
    }
   ],
   "source": [
    "dehb = DEHB(\n",
    "    f=target_function,\n",
    "    dimensions=dimensions,\n",
    "    cs=cs,\n",
    "    min_budget=min_budget,\n",
    "    max_budget=max_budget,\n",
    "    output_path=\"./temp\",\n",
    "    n_workers=2\n",
    ")\n",
    "trajectory, runtime, history = dehb.run(\n",
    "    total_cost=20, verbose=False,\n",
    ")\n",
    "\n",
    "print(dehb.get_incumbents())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Configuration:\n",
      "  x0, Value: 4.610766436763522\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(dehb.vector_to_configspace(dehb.inc_config))  # config as ConfigSpace configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.007774399252232556 0.007774399252232556\n",
      "Configuration:\n",
      "  x0, Value: 4.610766436763522\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(trajectory[-1], dehb.inc_score)\n",
    "print(dehb.vector_to_configspace(dehb.inc_config))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "### Conclusion\n",
    "\n",
    "As detailed above, the problem definition needs to be input to DEHB as the following information:\n",
    "* the *target_function* (`f`) that is the primary black-box function to optimize\n",
    "* the fidelity range of `min_budget` and `max_budget` that allows the cheaper, faster gray-box optimization of `f`\n",
    "* the search space or the input domain of the function `f`, that can be represented as a `ConfigSpace` object and passed to DEHB at initialization\n",
    "\n",
    "\n",
    "Following which, DEHB can be run for any amount of practical real-world budget. It can be run for either:\n",
    "* a total amount of actual wallclock time, example one day (~86400 seconds), or\n",
    "* a total number of function evaluations, or the number of times we want the black-box to be accessed for evaluation, across all fidelities\n",
    "* the total number of *brackets* we want to run the DEHB algorithm for\n",
    "\n",
    "DEHB will terminate once its chosen runtime budget is exhausted, and report the incumbent found. DEHB, as an *anytime* algorithm, constantly writes to disk a lightweight `json` file with the best found configuration and its score seen till that point."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dask",
   "language": "python",
   "name": "dask"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}