examples/01.2_Optimizing_RandomForest_using_Ask_Tell.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimizing RandomForest using the Ask & Tell interface\n",
    "\n",
    "This notebook aims to build on the template from `00_interfacing_DEHB` and use it on an actual problem, to optimize the hyperparameters of a Random Forest model, for a dataset. Here we use DEHB with the built-in ask and tell functionality.\n",
    "\n",
    "Additional requirements:\n",
    "* scikit-learn>=0.24\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import numpy as np\n",
    "import warnings\n",
    "\n",
    "\n",
    "seed = 123\n",
    "np.random.seed(seed)\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The problem defined here is to optimize a [Random Forest model](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html), on any given dataset, using DEHB. The hyperparameters chosen to be optimized are:\n",
    "* `max_depth`\n",
    "* `min_samples_split`\n",
    "* `max_features`\n",
    "* `min_samples_leaf`\n",
    "while the `n_estimators` hyperparameter to the Random Forest is chosen to be a fidelity parameter instead. Lesser number of trees ($<10$) in the Random Forest may not allow adequate ensembling for the grouped prediction to be significantly better than the individual tree predictions. Whereas a large number of trees (~$100$) often give accurate predictions but is naturally slower to train and predict on account of more trees to train. Therefore, a smaller `n_estimators` can be used as a cheaper approximation of the actual fidelity of `n_estimators=100`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining fidelity range"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_fidelity, max_fidelity = 2, 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the remaining $4$ hyperparameters, the search space can be created as a `ConfigSpace` object, with the domain of individual parameters defined."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating search space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ConfigSpace as CS\n",
    "\n",
    "\n",
    "def create_search_space(seed=123):\n",
    "    \"\"\"Parameter space to be optimized --- contains the hyperparameters\n",
    "    \"\"\"\n",
    "    cs = CS.ConfigurationSpace(seed=seed)\n",
    "\n",
    "    cs.add_hyperparameters([\n",
    "        CS.UniformIntegerHyperparameter(\n",
    "            'max_depth', lower=1, upper=15, default_value=2, log=False\n",
    "        ),\n",
    "        CS.UniformIntegerHyperparameter(\n",
    "            'min_samples_split', lower=2, upper=128, default_value=2, log=True\n",
    "        ),\n",
    "        CS.UniformFloatHyperparameter(\n",
    "            'max_features', lower=0.1, upper=0.9, default_value=0.5, log=False\n",
    "        ),\n",
    "        CS.UniformIntegerHyperparameter(\n",
    "            'min_samples_leaf', lower=1, upper=64, default_value=1, log=True\n",
    "        ),\n",
    "    ])\n",
    "    return cs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Configuration space object:\n",
      "  Hyperparameters:\n",
      "    max_depth, Type: UniformInteger, Range: [1, 15], Default: 2\n",
      "    max_features, Type: UniformFloat, Range: [0.1, 0.9], Default: 0.5\n",
      "    min_samples_leaf, Type: UniformInteger, Range: [1, 64], Default: 1, on log-scale\n",
      "    min_samples_split, Type: UniformInteger, Range: [2, 128], Default: 2, on log-scale\n",
      "\n"
     ]
    }
   ],
   "source": [
    "cs = create_search_space(seed)\n",
    "print(cs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dimensionality of search space: 4\n"
     ]
    }
   ],
   "source": [
    "dimensions = len(cs.get_hyperparameters())\n",
    "print(\"Dimensionality of search space: {}\".format(dimensions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the primary black/gray-box interface to the Random Forest model needs to be built for DEHB to query. As given in the `00_interfacing_DEHB` notebook, this function will have a signature akin to: `target_function(config, fidelity)`, and return a two-element tuple of the `score` and `cost`. It must be noted that DEHB **minimizes** and therefore the `score` being returned by this `target_function` should account for it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, the target function trains a Random Forest model on a dataset. We load a dataset here and maintain a fixed, train-validation-test split for one complete run. Multiple DEHB runs can therefore optimize on the same validation split, and evaluate final performance on the same test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating target function to optimize (2 parts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1 ) Preparing dataset and splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris, load_digits, load_wine\n",
    "\n",
    "\n",
    "classification = {\"iris\": load_iris, \"digits\": load_digits, \"wine\": load_wine}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "def prepare_dataset(model_type=\"classification\", dataset=None):\n",
    "\n",
    "    if model_type == \"classification\":\n",
    "        if dataset is None:\n",
    "            dataset = np.random.choice(list(classification.keys())) \n",
    "        _data = classification[dataset]()\n",
    "    else:\n",
    "        if dataset is None:\n",
    "            dataset = np.random.choice(list(regression.keys()))\n",
    "        _data = regression[dataset]()\n",
    "\n",
    "    train_X, rest_X, train_y, rest_y = train_test_split(\n",
    "      _data.get(\"data\"), \n",
    "      _data.get(\"target\"), \n",
    "      train_size=0.7, \n",
    "      shuffle=True, \n",
    "      random_state=seed\n",
    "    )\n",
    "    \n",
    "    # 10% test and 20% validation data\n",
    "    valid_X, test_X, valid_y, test_y = train_test_split(\n",
    "      rest_X, rest_y,\n",
    "      test_size=0.3333, \n",
    "      shuffle=True, \n",
    "      random_state=seed\n",
    "    )\n",
    "    return train_X, train_y, valid_X, valid_y, test_X, test_y, dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "wine\n",
      "Train size: (124, 13)\n",
      "Valid size: (36, 13)\n",
      "Test size: (18, 13)\n"
     ]
    }
   ],
   "source": [
    "train_X, train_y, valid_X, valid_y, test_X, test_y, dataset = \\\n",
    "    prepare_dataset(model_type=\"classification\")\n",
    "\n",
    "print(dataset)\n",
    "print(\"Train size: {}\\nValid size: {}\\nTest size: {}\".format(\n",
    "    train_X.shape, valid_X.shape, test_X.shape\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2 ) Function interface with DEHB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import accuracy_score, make_scorer\n",
    "\n",
    "\n",
    "accuracy_scorer = make_scorer(accuracy_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def target_function(config, fidelity, **kwargs):\n",
    "    # Extracting support information\n",
    "    seed = kwargs[\"seed\"]\n",
    "    train_X = kwargs[\"train_X\"]\n",
    "    train_y = kwargs[\"train_y\"]\n",
    "    valid_X = kwargs[\"valid_X\"]\n",
    "    valid_y = kwargs[\"valid_y\"]\n",
    "    max_fidelity = kwargs[\"max_fidelity\"]\n",
    "    \n",
    "    if fidelity is None:\n",
    "        fidelity = max_fidelity\n",
    "    \n",
    "    start = time.time()\n",
    "    # Building model \n",
    "    model = RandomForestClassifier(\n",
    "        **config.get_dictionary(),\n",
    "        n_estimators=int(fidelity),\n",
    "        bootstrap=True,\n",
    "        random_state=seed,\n",
    "    )\n",
    "    # Training the model on the complete training set\n",
    "    model.fit(train_X, train_y)\n",
    "    \n",
    "    # Evaluating the model on the validation set\n",
    "    valid_accuracy = accuracy_scorer(model, valid_X, valid_y)\n",
    "    cost = time.time() - start\n",
    "    \n",
    "    # Evaluating the model on the test set as additional info\n",
    "    test_accuracy = accuracy_scorer(model, test_X, test_y)\n",
    "    \n",
    "    result = {\n",
    "        \"fitness\": -valid_accuracy,  # DE/DEHB minimizes\n",
    "        \"cost\": cost,\n",
    "        \"info\": {\n",
    "            \"test_score\": test_accuracy,\n",
    "            \"fidelity\": fidelity\n",
    "        }\n",
    "    }\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have all components to define the problem to be optimized. DEHB can be initialized using all these information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running DEHB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dehb import DEHB\n",
    "\n",
    "dehb = DEHB(\n",
    "    f=target_function, # Here we do not need to necessarily specify the target function, but it can still be useful to call 'run' later.\n",
    "    cs=cs, \n",
    "    dimensions=dimensions, \n",
    "    min_fidelity=min_fidelity, \n",
    "    max_fidelity=max_fidelity,\n",
    "    n_workers=1,\n",
    "    output_path=\"./temp\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_function_evals = 50\n",
    "\n",
    "for _ in range(n_function_evals):\n",
    "    # Ask for the job_info, including the configuration to run and the fidelity\n",
    "    job_info = dehb.ask()\n",
    "\n",
    "    # Evaluate the configuration on the given fidelity. Here you are free to use\n",
    "    # any technique to compute the result. This job could e.g. be forwarded to\n",
    "    # a worker on your cluster (Which is not required to use Dask).\n",
    "    # The results dict has to contain the keys \"cost\" and \"fitness\" with an additional \"info\"\n",
    "    # dict for additional, user-specific data.\n",
    "    res = target_function(job_info[\"config\"], job_info[\"fidelity\"],\n",
    "                          # parameters as **kwargs in target_function\n",
    "                          seed=123,\n",
    "                          train_X=train_X,\n",
    "                          train_y=train_y,\n",
    "                          valid_X=valid_X,\n",
    "                          valid_y=valid_y,\n",
    "                          max_fidelity=dehb.max_fidelity)\n",
    "    \n",
    "    # When the evaluation is done, report the results back to the DEHB controller.\n",
    "    dehb.tell(job_info, res)\n",
    "\n",
    "trajectory = dehb.traj\n",
    "runtime = dehb.runtime\n",
    "history = dehb.history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50 50 50\n",
      "\n",
      "Last evaluated configuration, \n",
      "Configuration(values={\n",
      "  'max_depth': 12,\n",
      "  'max_features': 0.6422911372304849,\n",
      "  'min_samples_leaf': 9,\n",
      "  'min_samples_split': 8,\n",
      "})got a score of -0.9722222222222222, was evaluated at a fidelity of 16.67 and took 0.015 seconds to run.\n",
      "The additional info attached: {'test_score': 1.0, 'fidelity': 16.666666666666664}\n",
      "\n",
      "Best evaluated configuration, \n",
      "Configuration(values={\n",
      "  'max_depth': 12,\n",
      "  'max_features': 0.6422911372304849,\n",
      "  'min_samples_leaf': 8,\n",
      "  'min_samples_split': 8,\n",
      "}) got an accuracy of 1.0 on the test set.\n"
     ]
    }
   ],
   "source": [
    "print(len(trajectory), len(runtime), len(history), end=\"\\n\\n\")\n",
    "\n",
    "# Last recorded function evaluation\n",
    "last_eval = history[-1]\n",
    "config_id, config, score, cost, fidelity, _info = last_eval\n",
    "\n",
    "print(\"Last evaluated configuration, \")\n",
    "print(dehb.vector_to_configspace(config), end=\"\")\n",
    "print(\"got a score of {}, was evaluated at a fidelity of {:.2f} and \"\n",
    "      \"took {:.3f} seconds to run.\".format(score, fidelity, cost))\n",
    "print(\"The additional info attached: {}\".format(_info))\n",
    "\n",
    "print()\n",
    "print(\"Best evaluated configuration, \")\n",
    "\n",
    "best_config = dehb.vector_to_configspace(dehb.inc_config)\n",
    "\n",
    "# Creating a model using the best configuration found\n",
    "model = RandomForestClassifier(\n",
    "      **best_config.get_dictionary(),\n",
    "      n_estimators=int(max_fidelity),\n",
    "      bootstrap=True,\n",
    "      random_state=seed,\n",
    ")\n",
    "# Training the model on the complete training set\n",
    "model.fit(\n",
    "      np.concatenate((train_X, valid_X)), \n",
    "      np.concatenate((train_y, valid_y))\n",
    ")\n",
    "# Evaluating the model on the held-out test set\n",
    "test_accuracy = accuracy_scorer(model, test_X, test_y)\n",
    "\n",
    "print(f\"{best_config} got an accuracy of {test_accuracy} on the test set.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running DEHB for 50 function evaluations using the ask and tell interface, we can still call the `run` function in order keep optimizing without specifically using ask and tell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-01-22 12:11:18.288\u001b[0m | \u001b[31m\u001b[1mERROR   \u001b[0m | \u001b[36m__main__\u001b[0m:\u001b[36m<module>\u001b[0m:\u001b[36m2\u001b[0m - \u001b[31m\u001b[1mAn error has been caught in function '<module>', process 'MainProcess' (24895), thread 'MainThread' (140312493195648):\u001b[0m\n",
      "\u001b[33m\u001b[1mTraceback (most recent call last):\u001b[0m\n",
      "\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/runpy.py\", line 197, in _run_module_as_main\n",
      "    return _run_code(code, main_globals, None,\n",
      "           │         │     └ {'__name__': '__main__', '__doc__': 'Entry point for launching an IPython kernel.\\n\\nThis is separate from the ipykernel pack...\n",
      "           │         └ <code object <module> at 0x7f9d0b6de240, file \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel_launche...\n",
      "           └ <function _run_code at 0x7f9d0b6dba60>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/runpy.py\", line 87, in _run_code\n",
      "    exec(code, run_globals)\n",
      "         │     └ {'__name__': '__main__', '__doc__': 'Entry point for launching an IPython kernel.\\n\\nThis is separate from the ipykernel pack...\n",
      "         └ <code object <module> at 0x7f9d0b6de240, file \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel_launche...\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel_launcher.py\", line 17, in <module>\n",
      "    app.launch_new_instance()\n",
      "    │   └ <bound method Application.launch_instance of <class 'ipykernel.kernelapp.IPKernelApp'>>\n",
      "    └ <module 'ipykernel.kernelapp' from '/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/kernelapp.py'>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/traitlets/config/application.py\", line 1075, in launch_instance\n",
      "    app.start()\n",
      "    │   └ <function IPKernelApp.start at 0x7f9d07ba11f0>\n",
      "    └ <ipykernel.kernelapp.IPKernelApp object at 0x7f9d0c1036d0>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/kernelapp.py\", line 739, in start\n",
      "    self.io_loop.start()\n",
      "    │    │       └ <function BaseAsyncIOLoop.start at 0x7f9d07ba1ee0>\n",
      "    │    └ <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7f9d068f51f0>\n",
      "    └ <ipykernel.kernelapp.IPKernelApp object at 0x7f9d0c1036d0>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/tornado/platform/asyncio.py\", line 205, in start\n",
      "    self.asyncio_loop.run_forever()\n",
      "    │    │            └ <function BaseEventLoop.run_forever at 0x7f9d0980ee50>\n",
      "    │    └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n",
      "    └ <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7f9d068f51f0>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/asyncio/base_events.py\", line 601, in run_forever\n",
      "    self._run_once()\n",
      "    │    └ <function BaseEventLoop._run_once at 0x7f9d098129d0>\n",
      "    └ <_UnixSelectorEventLoop running=True closed=False debug=False>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/asyncio/base_events.py\", line 1905, in _run_once\n",
      "    handle._run()\n",
      "    │      └ <function Handle._run at 0x7f9d098c5e50>\n",
      "    └ <Handle <TaskWakeupMethWrapper object at 0x7f9cdc3bc190>(<Future finis...460>, ...],))>)>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/asyncio/events.py\", line 80, in _run\n",
      "    self._context.run(self._callback, *self._args)\n",
      "    │    │            │    │           │    └ <member '_args' of 'Handle' objects>\n",
      "    │    │            │    │           └ <Handle <TaskWakeupMethWrapper object at 0x7f9cdc3bc190>(<Future finis...460>, ...],))>)>\n",
      "    │    │            │    └ <member '_callback' of 'Handle' objects>\n",
      "    │    │            └ <Handle <TaskWakeupMethWrapper object at 0x7f9cdc3bc190>(<Future finis...460>, ...],))>)>\n",
      "    │    └ <member '_context' of 'Handle' objects>\n",
      "    └ <Handle <TaskWakeupMethWrapper object at 0x7f9cdc3bc190>(<Future finis...460>, ...],))>)>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/kernelbase.py\", line 542, in dispatch_queue\n",
      "    await self.process_one()\n",
      "          │    └ <function Kernel.process_one at 0x7f9d08091550>\n",
      "          └ <ipykernel.ipkernel.IPythonKernel object at 0x7f9d068f5850>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/kernelbase.py\", line 531, in process_one\n",
      "    await dispatch(*args)\n",
      "          │         └ ([<zmq.sugar.frame.Frame object at 0x7f9d07b47510>, <zmq.sugar.frame.Frame object at 0x7f9d040ad5c0>, <zmq.sugar.frame.Frame ...\n",
      "          └ <bound method Kernel.dispatch_shell of <ipykernel.ipkernel.IPythonKernel object at 0x7f9d068f5850>>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/kernelbase.py\", line 437, in dispatch_shell\n",
      "    await result\n",
      "          └ <coroutine object IPythonKernel.execute_request at 0x7f9c9053e940>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/ipkernel.py\", line 359, in execute_request\n",
      "    await super().execute_request(stream, ident, parent)\n",
      "                                  │       │      └ {'header': {'date': datetime.datetime(2024, 1, 22, 11, 11, 8, 246000, tzinfo=tzutc()), 'msg_id': '07608e16-efe5-484b-a3c8-8d9...\n",
      "                                  │       └ [b'5aebe739-ad41-4e49-bdd0-f4677ebbb3ca']\n",
      "                                  └ <zmq.eventloop.zmqstream.ZMQStream object at 0x7f9d068e8f10>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/kernelbase.py\", line 775, in execute_request\n",
      "    reply_content = await reply_content\n",
      "                          └ <coroutine object IPythonKernel.do_execute at 0x7f9d07b5f840>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/ipkernel.py\", line 446, in do_execute\n",
      "    res = shell.run_cell(\n",
      "          │     └ <function ZMQInteractiveShell.run_cell at 0x7f9d07b8f700>\n",
      "          └ <ipykernel.zmqshell.ZMQInteractiveShell object at 0x7f9d068f5dc0>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/ipykernel/zmqshell.py\", line 549, in run_cell\n",
      "    return super().run_cell(*args, **kwargs)\n",
      "                             │       └ {'store_history': True, 'silent': False, 'cell_id': '25acb834-4792-496e-b228-a6ef90905ea9'}\n",
      "                             └ ('# Continuing the ask/tell run of DEHB optimization for another 10s\\ntrajectory, runtime, history = dehb.run(\\n    total_cos...\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/IPython/core/interactiveshell.py\", line 3048, in run_cell\n",
      "    result = self._run_cell(\n",
      "             │    └ <function InteractiveShell._run_cell at 0x7f9d08ae0160>\n",
      "             └ <ipykernel.zmqshell.ZMQInteractiveShell object at 0x7f9d068f5dc0>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/IPython/core/interactiveshell.py\", line 3103, in _run_cell\n",
      "    result = runner(coro)\n",
      "             │      └ <coroutine object InteractiveShell.run_cell_async at 0x7f9d07b5fbc0>\n",
      "             └ <function _pseudo_sync_runner at 0x7f9d08b4d9d0>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/IPython/core/async_helpers.py\", line 129, in _pseudo_sync_runner\n",
      "    coro.send(None)\n",
      "    │    └ <method 'send' of 'coroutine' objects>\n",
      "    └ <coroutine object InteractiveShell.run_cell_async at 0x7f9d07b5fbc0>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/IPython/core/interactiveshell.py\", line 3308, in run_cell_async\n",
      "    has_raised = await self.run_ast_nodes(code_ast.body, cell_name,\n",
      "                       │    │             │        │     └ '/tmp/ipykernel_24895/2702971036.py'\n",
      "                       │    │             │        └ [<ast.Assign object at 0x7f9c8f1339d0>, <ast.Assign object at 0x7f9c8f1339a0>, <ast.Assign object at 0x7f9c8f133820>, <ast.Ex...\n",
      "                       │    │             └ <ast.Module object at 0x7f9c8f133b50>\n",
      "                       │    └ <function InteractiveShell.run_ast_nodes at 0x7f9d08ae0430>\n",
      "                       └ <ipykernel.zmqshell.ZMQInteractiveShell object at 0x7f9d068f5dc0>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/IPython/core/interactiveshell.py\", line 3490, in run_ast_nodes\n",
      "    if await self.run_code(code, result, async_=asy):\n",
      "             │    │        │     │              └ False\n",
      "             │    │        │     └ <ExecutionResult object at 7f9c8f141820, execution_count=14 error_before_exec=None error_in_exec=None info=<ExecutionInfo obj...\n",
      "             │    │        └ <code object <module> at 0x7f9c905735b0, file \"/tmp/ipykernel_24895/2702971036.py\", line 2>\n",
      "             │    └ <function InteractiveShell.run_code at 0x7f9d08ae04c0>\n",
      "             └ <ipykernel.zmqshell.ZMQInteractiveShell object at 0x7f9d068f5dc0>\n",
      "  File \"/home/fixja/miniconda3/envs/dehb/lib/python3.9/site-packages/IPython/core/interactiveshell.py\", line 3550, in run_code\n",
      "    exec(code_obj, self.user_global_ns, self.user_ns)\n",
      "         │         │    │               │    └ {'__name__': '__main__', '__doc__': 'Automatically created module for IPython interactive environment', '__package__': None, ...\n",
      "         │         │    │               └ <ipykernel.zmqshell.ZMQInteractiveShell object at 0x7f9d068f5dc0>\n",
      "         │         │    └ <property object at 0x7f9d08b5d090>\n",
      "         │         └ <ipykernel.zmqshell.ZMQInteractiveShell object at 0x7f9d068f5dc0>\n",
      "         └ <code object <module> at 0x7f9c905735b0, file \"/tmp/ipykernel_24895/2702971036.py\", line 2>\n",
      "\n",
      "> File \"\u001b[32m/tmp/ipykernel_24895/\u001b[0m\u001b[32m\u001b[1m2702971036.py\u001b[0m\", line \u001b[33m2\u001b[0m, in \u001b[35m<module>\u001b[0m\n",
      "    \u001b[1mtrajectory\u001b[0m\u001b[1m,\u001b[0m \u001b[1mruntime\u001b[0m\u001b[1m,\u001b[0m \u001b[1mhistory\u001b[0m \u001b[35m\u001b[1m=\u001b[0m \u001b[1mdehb\u001b[0m\u001b[35m\u001b[1m.\u001b[0m\u001b[1mrun\u001b[0m\u001b[1m(\u001b[0m\n",
      "    \u001b[36m│           │                  │    └ \u001b[0m\u001b[36m\u001b[1m<function DEHB.run at 0x7f9c8f964d30>\u001b[0m\n",
      "    \u001b[36m│           │                  └ \u001b[0m\u001b[36m\u001b[1m<dehb.optimizers.dehb.DEHB object at 0x7f9d040a55b0>\u001b[0m\n",
      "    \u001b[36m│           └ \u001b[0m\u001b[36m\u001b[1m[0.008121728897094727, 0.006032705307006836, 0.0056040287017822266, 0.006604433059692383, 0.005105018615722656, 0.00541806221...\u001b[0m\n",
      "    \u001b[36m└ \u001b[0m\u001b[36m\u001b[1m[-0.9722222222222222, -0.9722222222222222, -0.9722222222222222, -0.9722222222222222, -0.9722222222222222, -0.9722222222222222...\u001b[0m\n",
      "\n",
      "  File \"\u001b[32m/home/fixja/DEHB/src/dehb/optimizers/\u001b[0m\u001b[32m\u001b[1mdehb.py\u001b[0m\", line \u001b[33m923\u001b[0m, in \u001b[35mrun\u001b[0m\n",
      "    \u001b[1mself\u001b[0m\u001b[35m\u001b[1m.\u001b[0m\u001b[1mactive_brackets\u001b[0m\u001b[1m[\u001b[0m\u001b[34m\u001b[1m0\u001b[0m\u001b[1m]\u001b[0m\u001b[35m\u001b[1m.\u001b[0m\u001b[1mreset_waiting_jobs\u001b[0m\u001b[1m(\u001b[0m\u001b[1m)\u001b[0m\n",
      "    \u001b[36m│    └ \u001b[0m\u001b[36m\u001b[1m[]\u001b[0m\n",
      "    \u001b[36m└ \u001b[0m\u001b[36m\u001b[1m<dehb.optimizers.dehb.DEHB object at 0x7f9d040a55b0>\u001b[0m\n",
      "\n",
      "\u001b[31m\u001b[1mIndexError\u001b[0m:\u001b[1m list index out of range\u001b[0m\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "cannot unpack non-iterable NoneType object",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[14], line 2\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;66;03m# Continuing the ask/tell run of DEHB optimization for another 10s\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m trajectory, runtime, history \u001b[38;5;241m=\u001b[39m dehb\u001b[38;5;241m.\u001b[39mrun(\n\u001b[1;32m      3\u001b[0m     total_cost\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m10\u001b[39m,\n\u001b[1;32m      4\u001b[0m     verbose\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mFalse\u001b[39;00m,\n\u001b[1;32m      5\u001b[0m     save_intermediate\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mFalse\u001b[39;00m,\n\u001b[1;32m      6\u001b[0m     seed\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m123\u001b[39m,\n\u001b[1;32m      7\u001b[0m     train_X\u001b[38;5;241m=\u001b[39mtrain_X,\n\u001b[1;32m      8\u001b[0m     train_y\u001b[38;5;241m=\u001b[39mtrain_y,\n\u001b[1;32m      9\u001b[0m     valid_X\u001b[38;5;241m=\u001b[39mvalid_X,\n\u001b[1;32m     10\u001b[0m     valid_y\u001b[38;5;241m=\u001b[39mvalid_y,\n\u001b[1;32m     11\u001b[0m     max_fidelity\u001b[38;5;241m=\u001b[39mdehb\u001b[38;5;241m.\u001b[39mmax_fidelity\n\u001b[1;32m     12\u001b[0m )\n\u001b[1;32m     13\u001b[0m best_config \u001b[38;5;241m=\u001b[39m dehb\u001b[38;5;241m.\u001b[39mvector_to_configspace(dehb\u001b[38;5;241m.\u001b[39minc_config)\n\u001b[1;32m     15\u001b[0m \u001b[38;5;66;03m# Creating a model using the best configuration found\u001b[39;00m\n",
      "\u001b[0;31mTypeError\u001b[0m: cannot unpack non-iterable NoneType object"
     ]
    }
   ],
   "source": [
    "# Continuing the ask/tell run of DEHB optimization for another 10s\n",
    "trajectory, runtime, history = dehb.run(\n",
    "    total_cost=10,\n",
    "    seed=123,\n",
    "    train_X=train_X,\n",
    "    train_y=train_y,\n",
    "    valid_X=valid_X,\n",
    "    valid_y=valid_y,\n",
    "    max_fidelity=dehb.max_fidelity\n",
    ")\n",
    "best_config = dehb.vector_to_configspace(dehb.inc_config)\n",
    "\n",
    "# Creating a model using the best configuration found\n",
    "model = RandomForestClassifier(\n",
    "    **best_config.get_dictionary(),\n",
    "    n_estimators=int(max_fidelity),\n",
    "    bootstrap=True,\n",
    "    random_state=seed,\n",
    ")\n",
    "# Training the model on the complete training set\n",
    "model.fit(\n",
    "    np.concatenate((train_X, valid_X)), \n",
    "    np.concatenate((train_y, valid_y))\n",
    ")\n",
    "# Evaluating the model on the held-out test set\n",
    "test_accuracy = accuracy_scorer(model, test_X, test_y)\n",
    "\n",
    "\n",
    "print(len(trajectory), len(runtime), len(history), end=\"\\n\\n\")\n",
    "\n",
    "# Last recorded function evaluation\n",
    "last_eval = history[-1]\n",
    "config_id, config, score, cost, fidelity, _info = last_eval\n",
    "\n",
    "print(\"Last evaluated configuration, \")\n",
    "print(dehb.vector_to_configspace(config), end=\"\")\n",
    "print(\"got a score of {}, was evaluated at a fidelity of {:.2f} and \"\n",
    "      \"took {:.3f} seconds to run.\".format(score, fidelity, cost))\n",
    "print(\"The additional info attached: {}\".format(_info))\n",
    "\n",
    "print()\n",
    "print(\"Best evaluated configuration, \")\n",
    "print(f\"{best_config} got an accuracy of {test_accuracy} on the test set.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}