Lectures/Lecture_L2O_Project/presentation_l2o_project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "70972ef6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=./img/Brown_logo.svg width=30%>\n",
    "\n",
    "# Project 1: Learning to Choose Optimizers\n",
    "\n",
    "\n",
    "### Martin van der Schelling | <a href = \"mailto: martin_van_der_schelling@brown.edu\">martin_van_der_schelling@brown.edu</a>  | PhD candidate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "507e04ce",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Outline of today\n",
    "\n",
    "At the end of this lecture you have learned:\n",
    "\n",
    "* Why we need optimization algorithms\n",
    "* Why we need to make an informed decision on our optimizer choice\n",
    "* How we might get free lunch anyway!\n",
    "* How to get started on project 1: L2O\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e7cd91c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Optimization problems are everywhere\n",
    "\n",
    "Optimization is the process of finding the **best solution** or outcome from a set of possible options based on a certain **criteria**.\n",
    "\n",
    "<!-- * learning goal: understand that optimization is used in machine learning but also in other disciplines: -->\n",
    "\n",
    "<img src=./img/examples_opt.png width=60%>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61b371e5",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Different kind of optimization problems\n",
    "\n",
    "### Time complexity\n",
    "* *\"How much does the time to solve increase when we make the problem bigger?\"*\n",
    "* Time complexity is a measure of the **amount of time an algorithm takes as a function of the size of the input ($n$)**\n",
    "\n",
    "* This is often expressed using **\"Big O\" notation**: $\\mathcal{O}(n)$, $\\mathcal{O}(n \\; \\mathrm{log}(n))$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44aabfb5",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Why is this important?\n",
    "* Gives us a sense if the algorithm can solve this problem\n",
    "* It let us compare the **efficiency** of algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7438589",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=./img/time-complexity-examples.png width=40%>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1bcfdeb",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "What happens when we **scale the problem** and try to solve a optimization problem that has $\\mathcal{O}(2^n)$ ?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6424750",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**We run out of memory/time to solve the problem exactly**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ae51dc6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Well how do we solve those problems?\n",
    "\n",
    "🫱🏽‍🫲🏻 Making a compromise: Get a good enough solution within polynomial time complexity\n",
    "* Trade-off between **complexity** and **solution quality**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15ec2684",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Iterative procedure\n",
    "\n",
    "<img src=./img/schematic_optimization.png width=80%>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dc66818",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Step 1) Choose an **initial guess** ($\\mathbf{x}_0$)\n",
    "\n",
    "Step 2) Update your current solution with an **optimization algorithm**\n",
    "\n",
    " $\\mathbf{x}_{t+1} = \\mathbf{x}_t + \\omega$\n",
    " \n",
    "Step 3) Repeat step 2 until some **stopping criteria**\n",
    "\n",
    "Step 4) Take your optimized value and hope for the best :)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "822ce683",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## For now, you can treat the optimization as a black box\n",
    "\n",
    "*More in-depth information about 'opening the black-box' of optimization in a later lecture!*\n",
    "\n",
    "<img src=./img/opt_blackbox.png width=40%>\n",
    "\n",
    "Your next iteration $\\mathbf{x}_{t+1}$ can depend on:\n",
    "* The **current solution** ($\\mathbf{x}_{t}$) and **response** ($y_{t}$) that you have\n",
    "* (Optionally) other information like **gradients** or **history of evaluations** ($\\mathbf{X}_{0 .. t}$ and $\\mathbf{y}_{0..t}$)\n",
    "* The **choice** of optimization algorithm\n",
    "* Any **hyperparameters** of the optimizer (e.g. the learning rate $\\alpha$)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "709e9165",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## There are many, many different optimization algorithms ..\n",
    "\n",
    "Each field has its own collection of optimizers: \n",
    "\n",
    "| Engineering Application | Optimization Algorithms |\n",
    "|-------------------------|--------------------------|\n",
    "| Structural design      | Genetic algorithms, simulated annealing, gradient descent |\n",
    "| Topology optimization  | Method of moving Asymptotes (MMA), Interior-point line-search (IPOPT), Optimality Criteria (OC) |\n",
    "| Process control        | Linear programming, quadratic programming, nonlinear programming |\n",
    "| Supply chain management| Linear programming, mixed integer programming |\n",
    "| Machine learning       | Stochastic gradient descent (SGD), Adam, Conjugate gradient (CG) |\n",
    "| Computer vision        | Gradient descent, stochastic gradient descent (SGD), coordinate descent |\n",
    "| Robotics               | Trajectory optimization, model predictive control |\n",
    "| Protein docking        | Monte Carlo with minimization (MCM), conformational space annealing (CSA), particle swarm optimization (PSO) |\n",
    " \n",
    "Basically, every optimizer has an $\\omega$ operation of calculating the next iterate $\\mathbf{x}_{t+1}$\n",
    "\n",
    "For example: Gradient descent: \n",
    "\n",
    "$\\mathbf{x}_{t+1} = \\mathbf{x}_{t} - \\alpha \\cdot \\frac{dy}{dx}$\n",
    "\n",
    "$\\omega = - \\alpha \\cdot \\frac{dy}{dx}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffd9b511",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "But why so many? Why don't we have **one optimizer to rule them all**?\n",
    "\n",
    "<img src=./img/one_optimizer.png width=40%, aling='center'>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebf65d85",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ❌🥪 No Free Lunch Theorem [1]\n",
    "\n",
    "> \"Any elevated performance over one class of problems is offset by performance over another class.\"\n",
    "\n",
    "\n",
    "*[1] Wolpert, D. H., & Macready, W. G. (1997). No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation, 1(1), 67–82. https://doi.org/10.1109/4235.585893*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88586969",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In other words; some optimization algorithms work on specific problems, but might not work for others!\n",
    "\n",
    "In the limit: **no optimization algorithm will perform better than random search over the entire space of optimization problems**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45f616e5",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Example: gradient based optimizer\n",
    "<img src=./img/Adam.gif width=40%, align='right'>\n",
    "\n",
    "\n",
    "\n",
    "<img src=./img/Sphere.png width=20%, align='left'><img src=./img/Schwefel.png width=20%, align='left'>\n",
    "\n",
    "\n",
    "* Gradient based optimizers work well in convex landscapes\n",
    "* They fail when multiple local minima are involved!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20c9e6b1",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Optimizers are designed to **exploit differen problem characteristics** in order to gain an advantage over random search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ef08552",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## How do we choose an optimizer?\n",
    "\n",
    "* Choose one based on the **knowledge** you have about your problem\n",
    "* Or you can try a bunch of them out (architecture search)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5365d9c9",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "But what if we **learn our optimization choice from data?**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b13c7ddd",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Learning to Optimize (L2O)\n",
    "\n",
    "Adjust optimizer based on the response of the problem\n",
    "\n",
    "<img src=./img/l2o.png width=40%>\n",
    "\n",
    "*[2] Li, K., & Malik, J. (2017). Learning to optimize. 5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ef47296",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "From **constant** update step to **trainable model**:\n",
    "\n",
    "* 'Classic' optimizer: $\\mathbf{x}_{t+1} = \\mathbf{x}_t + \\omega$\n",
    "* Learning to optimize: $\\mathbf{x}_{t+1} = \\mathbf{x}_t + m(\\omega ; \\phi)$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02825f34",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "<!-- ### Classic optimizer -->\n",
    "\n",
    "<img src=./img/schematic_opt_chen2021.png width=45%, align='left'>\n",
    "\n",
    "<!-- ### Learning to optimize -->\n",
    "\n",
    "<img src=./img/schematic_l2o_chen2021.png width=45%, align='right'>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e896d438",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    " \n",
    "*T. Chen, X. Chen, W. Chen, H. Heaton, J. Liu, Z. Wang, and W. Yin, Learning to Optimize: A Primer and A Benchmark, 2021 (2021)*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21de46f0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Project 1: L2O\n",
    "\n",
    "**Provided data and resources**: Two datasets with the training dynamics of a set of benchmark funcitons, optimized with a variety of hand-engineered optimizers: \n",
    "\n",
    "**Aim of the project**: Design a general *data-driven optimization strategy* with machine learning that competes with classic hand-engineered optimizers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb4ba43f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### How is the dataset generated?\n",
    "\n",
    "... with `f3dasm` of course ! 🌟🌟🌟🌟🌟\n",
    "\n",
    "<img src=\"./img/blocks_project.png\" width=\"100%\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f657be6d",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-10-27 12:01:12.239326: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
      "2023-10-27 12:01:12,845 - f3dasm - INFO - Imported f3dasm (version: 1.4.3)\n"
     ]
    }
   ],
   "source": [
    "from f3dasm import ExperimentData\n",
    "from f3dasm.design import Domain, make_nd_continuous_domain\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "235216aa",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "First, we create a `Domain` object with continuous input variables bounded by $[0.0, 1.0]^{d}$, where $d$ is the number of input parameters (=`dimensionality`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "30260aca",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Domain(space={'x0': ContinuousParameter(lower_bound=0.0, upper_bound=1.0, log=False), 'x1': ContinuousParameter(lower_bound=0.0, upper_bound=1.0, log=False)})"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "domain = make_nd_continuous_domain(bounds=[[0.0, 1.0], [0.0, 1.0]], dimensionality=2)\n",
    "domain"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f2e89f5",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "*`make_nd_continuous_domain` is a helper function to create n-dimensional continuous design spaces*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b416ba0e",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Then, we sample $30$ initial points from the domain with Latin Hypercube sampling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "98642172",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "experimentdata = ExperimentData(domain)\n",
    "experimentdata.sample(sampler='latin', n_samples=30, seed=2023)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d631b6cd",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We evaluate the samples with one of the benchmark functions (in this case the `Ackley` function)\n",
    "We can provide the following options:\n",
    "* `noise`: Gaussian noise with a certain standard deviation\n",
    "* `seed`: If the function is noisy, seed for the pseudo random-number-generator\n",
    "* `scale_bounds`: Scaling the function to some other boundaries, in this case the box-constraints of the `Domain`\n",
    "\n",
    "> Note: Each benchmark-function is **off-set by a random vector**. As most of the benchmark functions have their global minimum in the center of the design space, we want to avoid optimizers to \"cheat\" their way to the optimum by immediately jumping to the middle.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c66e6abf",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This is how a 2D-representation of these analytical functions looks like:\n",
    "\n",
    "<img src=./img/func_cropped.gif width=30%>\n",
    "\n",
    "Each of the generated functions have **labels** that describe the **characteristics** of the loss-landscape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c79f07c2",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "experimentdata.evaluate(data_generator='Ackley', \n",
    "                        kwargs={'seed': 2023, 'noise': 0.0, 'scale_bounds': domain.get_bounds()})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bf3971f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We **optimize** the benchmark function with an optimizer for $2000$ iterations (our `budget`) with the default hyper-parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "42dc08eb",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "experimentdata.optimize(data_generator='Ackley', optimizer='CMAES', iterations=2000, hyperparameters={'seed': 2023},\n",
    "                       kwargs={'seed': 2023, 'noise': 0.0, 'scale_bounds': domain.get_bounds()})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ad81d881",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>jobs</th>\n",
       "      <th colspan=\"2\" halign=\"left\">input</th>\n",
       "      <th>output</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>x0</th>\n",
       "      <th>x1</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>finished</td>\n",
       "      <td>0.181879</td>\n",
       "      <td>0.857659</td>\n",
       "      <td>21.650953</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>finished</td>\n",
       "      <td>0.377353</td>\n",
       "      <td>0.790722</td>\n",
       "      <td>21.957123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>finished</td>\n",
       "      <td>0.945234</td>\n",
       "      <td>0.643587</td>\n",
       "      <td>21.709284</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>finished</td>\n",
       "      <td>0.629493</td>\n",
       "      <td>0.603660</td>\n",
       "      <td>21.623877</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>finished</td>\n",
       "      <td>0.313033</td>\n",
       "      <td>0.238736</td>\n",
       "      <td>20.218512</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2025</th>\n",
       "      <td>finished</td>\n",
       "      <td>0.588052</td>\n",
       "      <td>0.126597</td>\n",
       "      <td>0.000129</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2026</th>\n",
       "      <td>finished</td>\n",
       "      <td>0.588052</td>\n",
       "      <td>0.126596</td>\n",
       "      <td>0.000064</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2027</th>\n",
       "      <td>finished</td>\n",
       "      <td>0.588052</td>\n",
       "      <td>0.126596</td>\n",
       "      <td>0.000066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2028</th>\n",
       "      <td>finished</td>\n",
       "      <td>0.588052</td>\n",
       "      <td>0.126596</td>\n",
       "      <td>0.000024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2029</th>\n",
       "      <td>finished</td>\n",
       "      <td>0.588052</td>\n",
       "      <td>0.126596</td>\n",
       "      <td>0.000076</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2030 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "<f3dasm._src.experimentdata.experimentdata.ExperimentData at 0x7f254f04e280>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experimentdata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3db74a58",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We redo this optimization for **different initial conditions** ($x_{0}$) and save the **entire history of iterations**.\n",
    "\n",
    "\n",
    "We run this optimization process with a **set of different optimizers**\n",
    "* Adam\n",
    "* Covariance matrix adaptation evolution strategy (CMAES)\n",
    "* Particle Swarm Optimization (PSO)\n",
    "* Limited-memory BFGS with box constraints (L-BFGS-B)\n",
    "* Random Search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7693a0f4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Each experiment is one instantiation of a benchmark function that is optimized for multiple realizations with a set of optimizers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "55f29c22",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>jobs</th>\n",
       "      <th colspan=\"5\" halign=\"left\">input</th>\n",
       "      <th colspan=\"2\" halign=\"left\">output</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>budget</th>\n",
       "      <th>dimensionality</th>\n",
       "      <th>function_name</th>\n",
       "      <th>noise</th>\n",
       "      <th>seed</th>\n",
       "      <th>path_raw</th>\n",
       "      <th>path_post</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2198</th>\n",
       "      <td>finished</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>2</td>\n",
       "      <td>Thevenot</td>\n",
       "      <td>0.1</td>\n",
       "      <td>100895</td>\n",
       "      <td>raw/2198.nc</td>\n",
       "      <td>post/2198.nc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1478</th>\n",
       "      <td>finished</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>2</td>\n",
       "      <td>Quartic</td>\n",
       "      <td>0.1</td>\n",
       "      <td>91077</td>\n",
       "      <td>raw/1478.nc</td>\n",
       "      <td>post/1478.nc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>130</th>\n",
       "      <td>finished</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>2</td>\n",
       "      <td>Levy</td>\n",
       "      <td>0.0</td>\n",
       "      <td>554703</td>\n",
       "      <td>raw/130.nc</td>\n",
       "      <td>post/130.nc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2951</th>\n",
       "      <td>finished</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>100</td>\n",
       "      <td>Shubert</td>\n",
       "      <td>0.0</td>\n",
       "      <td>152730</td>\n",
       "      <td>raw/2951.nc</td>\n",
       "      <td>post/2951.nc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2547</th>\n",
       "      <td>finished</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>2</td>\n",
       "      <td>Schwefel2_20</td>\n",
       "      <td>0.1</td>\n",
       "      <td>204304</td>\n",
       "      <td>raw/2547.nc</td>\n",
       "      <td>post/2547.nc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1423</th>\n",
       "      <td>finished</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>2</td>\n",
       "      <td>XinSheYang</td>\n",
       "      <td>0.0</td>\n",
       "      <td>328933</td>\n",
       "      <td>raw/1423.nc</td>\n",
       "      <td>post/1423.nc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>713</th>\n",
       "      <td>finished</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>10</td>\n",
       "      <td>Ackley</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1357</td>\n",
       "      <td>raw/713.nc</td>\n",
       "      <td>post/713.nc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1954</th>\n",
       "      <td>finished</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>10</td>\n",
       "      <td>Powell</td>\n",
       "      <td>0.0</td>\n",
       "      <td>276639</td>\n",
       "      <td>raw/1954.nc</td>\n",
       "      <td>post/1954.nc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>691</th>\n",
       "      <td>finished</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>10</td>\n",
       "      <td>Ackley</td>\n",
       "      <td>0.0</td>\n",
       "      <td>504520</td>\n",
       "      <td>raw/691.nc</td>\n",
       "      <td>post/691.nc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>280</th>\n",
       "      <td>finished</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>100</td>\n",
       "      <td>StyblinskiTang</td>\n",
       "      <td>0.1</td>\n",
       "      <td>874591</td>\n",
       "      <td>raw/280.nc</td>\n",
       "      <td>post/280.nc</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "<f3dasm._src.experimentdata.experimentdata.ExperimentData at 0x7f254efe1e20>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "small_dataset = ExperimentData.from_file('/home/martin/Documents/GitHub/3dasm_course/Projects/L2O/STUFF_NOT_TO_UPLOAD/data/small_dataset/small_dataset')\n",
    "small_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f964b56",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The data for each experiment is stored in two ways:\n",
    "* `raw`: file with the **entire history of iterations** for multiple realization and optimizers\n",
    "* `post`: file with already post-processed data containing a **performance metric** and the **problem-specific characteristics**\n",
    "\n",
    "The datasets are described in more detail in the [assignment description](https://github.com/bessagroup/3dasm_course/tree/main/Projects). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52e49482",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 👋 Say hi to the multidimensional pandas: `xarray`:\n",
    "\n",
    "<img src=\"./img/hi_xarray.png\" title=\"hi xarray\" width=\"30%\" align=\"right\">\n",
    "<img src=\"./img/datastructure.png\" width=\"60%\" align=\"left\">\n",
    "\n",
    "* `pandas` is great when working with two coordinates (rows and columns) \n",
    "* However, when you have structured data that uses more coordinates it becomes complicated ..\n",
    "* `xarray`: a data-analysis library for multi-coordinate data with a similar API to `pandas`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d5e6989b",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import xarray as xr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3231d649",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "On the [3dasm GitHub page](https://github.com/bessagroup/3dasm_course) you can download a Python file (`l20.py`) with functions that help you on the project:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9e925172",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "# Insert the helper functions file in the system path so that Python can import it\n",
    "sys.path.insert(0, '/home/martin/Documents/GitHub/3dasm_course/Projects/L2O/')\n",
    "import l2o"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30bc21bc",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As an example; we can inspect the `raw` data of experiment \\#280 of the experiment from the small dataset with the `open_one_dataset_raw()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "dd95d476",
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><svg style=\"position: absolute; width: 0; height: 0; overflow: hidden\">\n",
       "<defs>\n",
       "<symbol id=\"icon-database\" viewBox=\"0 0 32 32\">\n",
       "<path d=\"M16 0c-8.837 0-16 2.239-16 5v4c0 2.761 7.163 5 16 5s16-2.239 16-5v-4c0-2.761-7.163-5-16-5z\"></path>\n",
       "<path d=\"M16 17c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z\"></path>\n",
       "<path d=\"M16 26c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z\"></path>\n",
       "</symbol>\n",
       "<symbol id=\"icon-file-text2\" viewBox=\"0 0 32 32\">\n",
       "<path d=\"M28.681 7.159c-0.694-0.947-1.662-2.053-2.724-3.116s-2.169-2.030-3.116-2.724c-1.612-1.182-2.393-1.319-2.841-1.319h-15.5c-1.378 0-2.5 1.121-2.5 2.5v27c0 1.378 1.122 2.5 2.5 2.5h23c1.378 0 2.5-1.122 2.5-2.5v-19.5c0-0.448-0.137-1.23-1.319-2.841zM24.543 5.457c0.959 0.959 1.712 1.825 2.268 2.543h-4.811v-4.811c0.718 0.556 1.584 1.309 2.543 2.268zM28 29.5c0 0.271-0.229 0.5-0.5 0.5h-23c-0.271 0-0.5-0.229-0.5-0.5v-27c0-0.271 0.229-0.5 0.5-0.5 0 0 15.499-0 15.5 0v7c0 0.552 0.448 1 1 1h7v19.5z\"></path>\n",
       "<path d=\"M23 26h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
       "<path d=\"M23 22h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
       "<path d=\"M23 18h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z\"></path>\n",
       "</symbol>\n",
       "</defs>\n",
       "</svg>\n",
       "<style>/* CSS stylesheet for displaying xarray objects in jupyterlab.\n",
       " *\n",
       " */\n",
       "\n",
       ":root {\n",
       "  --xr-font-color0: var(--jp-content-font-color0, rgba(0, 0, 0, 1));\n",
       "  --xr-font-color2: var(--jp-content-font-color2, rgba(0, 0, 0, 0.54));\n",
       "  --xr-font-color3: var(--jp-content-font-color3, rgba(0, 0, 0, 0.38));\n",
       "  --xr-border-color: var(--jp-border-color2, #e0e0e0);\n",
       "  --xr-disabled-color: var(--jp-layout-color3, #bdbdbd);\n",
       "  --xr-background-color: var(--jp-layout-color0, white);\n",
       "  --xr-background-color-row-even: var(--jp-layout-color1, white);\n",
       "  --xr-background-color-row-odd: var(--jp-layout-color2, #eeeeee);\n",
       "}\n",
       "\n",
       "html[theme=dark],\n",
       "body[data-theme=dark],\n",
       "body.vscode-dark {\n",
       "  --xr-font-color0: rgba(255, 255, 255, 1);\n",
       "  --xr-font-color2: rgba(255, 255, 255, 0.54);\n",
       "  --xr-font-color3: rgba(255, 255, 255, 0.38);\n",
       "  --xr-border-color: #1F1F1F;\n",
       "  --xr-disabled-color: #515151;\n",
       "  --xr-background-color: #111111;\n",
       "  --xr-background-color-row-even: #111111;\n",
       "  --xr-background-color-row-odd: #313131;\n",
       "}\n",
       "\n",
       ".xr-wrap {\n",
       "  display: block !important;\n",
       "  min-width: 300px;\n",
       "  max-width: 700px;\n",
       "}\n",
       "\n",
       ".xr-text-repr-fallback {\n",
       "  /* fallback to plain text repr when CSS is not injected (untrusted notebook) */\n",
       "  display: none;\n",
       "}\n",
       "\n",
       ".xr-header {\n",
       "  padding-top: 6px;\n",
       "  padding-bottom: 6px;\n",
       "  margin-bottom: 4px;\n",
       "  border-bottom: solid 1px var(--xr-border-color);\n",
       "}\n",
       "\n",
       ".xr-header > div,\n",
       ".xr-header > ul {\n",
       "  display: inline;\n",
       "  margin-top: 0;\n",
       "  margin-bottom: 0;\n",
       "}\n",
       "\n",
       ".xr-obj-type,\n",
       ".xr-array-name {\n",
       "  margin-left: 2px;\n",
       "  margin-right: 10px;\n",
       "}\n",
       "\n",
       ".xr-obj-type {\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-sections {\n",
       "  padding-left: 0 !important;\n",
       "  display: grid;\n",
       "  grid-template-columns: 150px auto auto 1fr 20px 20px;\n",
       "}\n",
       "\n",
       ".xr-section-item {\n",
       "  display: contents;\n",
       "}\n",
       "\n",
       ".xr-section-item input {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       ".xr-section-item input + label {\n",
       "  color: var(--xr-disabled-color);\n",
       "}\n",
       "\n",
       ".xr-section-item input:enabled + label {\n",
       "  cursor: pointer;\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-section-item input:enabled + label:hover {\n",
       "  color: var(--xr-font-color0);\n",
       "}\n",
       "\n",
       ".xr-section-summary {\n",
       "  grid-column: 1;\n",
       "  color: var(--xr-font-color2);\n",
       "  font-weight: 500;\n",
       "}\n",
       "\n",
       ".xr-section-summary > span {\n",
       "  display: inline-block;\n",
       "  padding-left: 0.5em;\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:disabled + label {\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-section-summary-in + label:before {\n",
       "  display: inline-block;\n",
       "  content: '►';\n",
       "  font-size: 11px;\n",
       "  width: 15px;\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:disabled + label:before {\n",
       "  color: var(--xr-disabled-color);\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:checked + label:before {\n",
       "  content: '▼';\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:checked + label > span {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       ".xr-section-summary,\n",
       ".xr-section-inline-details {\n",
       "  padding-top: 4px;\n",
       "  padding-bottom: 4px;\n",
       "}\n",
       "\n",
       ".xr-section-inline-details {\n",
       "  grid-column: 2 / -1;\n",
       "}\n",
       "\n",
       ".xr-section-details {\n",
       "  display: none;\n",
       "  grid-column: 1 / -1;\n",
       "  margin-bottom: 5px;\n",
       "}\n",
       "\n",
       ".xr-section-summary-in:checked ~ .xr-section-details {\n",
       "  display: contents;\n",
       "}\n",
       "\n",
       ".xr-array-wrap {\n",
       "  grid-column: 1 / -1;\n",
       "  display: grid;\n",
       "  grid-template-columns: 20px auto;\n",
       "}\n",
       "\n",
       ".xr-array-wrap > label {\n",
       "  grid-column: 1;\n",
       "  vertical-align: top;\n",
       "}\n",
       "\n",
       ".xr-preview {\n",
       "  color: var(--xr-font-color3);\n",
       "}\n",
       "\n",
       ".xr-array-preview,\n",
       ".xr-array-data {\n",
       "  padding: 0 5px !important;\n",
       "  grid-column: 2;\n",
       "}\n",
       "\n",
       ".xr-array-data,\n",
       ".xr-array-in:checked ~ .xr-array-preview {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       ".xr-array-in:checked ~ .xr-array-data,\n",
       ".xr-array-preview {\n",
       "  display: inline-block;\n",
       "}\n",
       "\n",
       ".xr-dim-list {\n",
       "  display: inline-block !important;\n",
       "  list-style: none;\n",
       "  padding: 0 !important;\n",
       "  margin: 0;\n",
       "}\n",
       "\n",
       ".xr-dim-list li {\n",
       "  display: inline-block;\n",
       "  padding: 0;\n",
       "  margin: 0;\n",
       "}\n",
       "\n",
       ".xr-dim-list:before {\n",
       "  content: '(';\n",
       "}\n",
       "\n",
       ".xr-dim-list:after {\n",
       "  content: ')';\n",
       "}\n",
       "\n",
       ".xr-dim-list li:not(:last-child):after {\n",
       "  content: ',';\n",
       "  padding-right: 5px;\n",
       "}\n",
       "\n",
       ".xr-has-index {\n",
       "  font-weight: bold;\n",
       "}\n",
       "\n",
       ".xr-var-list,\n",
       ".xr-var-item {\n",
       "  display: contents;\n",
       "}\n",
       "\n",
       ".xr-var-item > div,\n",
       ".xr-var-item label,\n",
       ".xr-var-item > .xr-var-name span {\n",
       "  background-color: var(--xr-background-color-row-even);\n",
       "  margin-bottom: 0;\n",
       "}\n",
       "\n",
       ".xr-var-item > .xr-var-name:hover span {\n",
       "  padding-right: 5px;\n",
       "}\n",
       "\n",
       ".xr-var-list > li:nth-child(odd) > div,\n",
       ".xr-var-list > li:nth-child(odd) > label,\n",
       ".xr-var-list > li:nth-child(odd) > .xr-var-name span {\n",
       "  background-color: var(--xr-background-color-row-odd);\n",
       "}\n",
       "\n",
       ".xr-var-name {\n",
       "  grid-column: 1;\n",
       "}\n",
       "\n",
       ".xr-var-dims {\n",
       "  grid-column: 2;\n",
       "}\n",
       "\n",
       ".xr-var-dtype {\n",
       "  grid-column: 3;\n",
       "  text-align: right;\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-var-preview {\n",
       "  grid-column: 4;\n",
       "}\n",
       "\n",
       ".xr-index-preview {\n",
       "  grid-column: 2 / 5;\n",
       "  color: var(--xr-font-color2);\n",
       "}\n",
       "\n",
       ".xr-var-name,\n",
       ".xr-var-dims,\n",
       ".xr-var-dtype,\n",
       ".xr-preview,\n",
       ".xr-attrs dt {\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "  padding-right: 10px;\n",
       "}\n",
       "\n",
       ".xr-var-name:hover,\n",
       ".xr-var-dims:hover,\n",
       ".xr-var-dtype:hover,\n",
       ".xr-attrs dt:hover {\n",
       "  overflow: visible;\n",
       "  width: auto;\n",
       "  z-index: 1;\n",
       "}\n",
       "\n",
       ".xr-var-attrs,\n",
       ".xr-var-data,\n",
       ".xr-index-data {\n",
       "  display: none;\n",
       "  background-color: var(--xr-background-color) !important;\n",
       "  padding-bottom: 5px !important;\n",
       "}\n",
       "\n",
       ".xr-var-attrs-in:checked ~ .xr-var-attrs,\n",
       ".xr-var-data-in:checked ~ .xr-var-data,\n",
       ".xr-index-data-in:checked ~ .xr-index-data {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       ".xr-var-data > table {\n",
       "  float: right;\n",
       "}\n",
       "\n",
       ".xr-var-name span,\n",
       ".xr-var-data,\n",
       ".xr-index-name div,\n",
       ".xr-index-data,\n",
       ".xr-attrs {\n",
       "  padding-left: 25px !important;\n",
       "}\n",
       "\n",
       ".xr-attrs,\n",
       ".xr-var-attrs,\n",
       ".xr-var-data,\n",
       ".xr-index-data {\n",
       "  grid-column: 1 / -1;\n",
       "}\n",
       "\n",
       "dl.xr-attrs {\n",
       "  padding: 0;\n",
       "  margin: 0;\n",
       "  display: grid;\n",
       "  grid-template-columns: 125px auto;\n",
       "}\n",
       "\n",
       ".xr-attrs dt,\n",
       ".xr-attrs dd {\n",
       "  padding: 0;\n",
       "  margin: 0;\n",
       "  float: left;\n",
       "  padding-right: 10px;\n",
       "  width: auto;\n",
       "}\n",
       "\n",
       ".xr-attrs dt {\n",
       "  font-weight: normal;\n",
       "  grid-column: 1;\n",
       "}\n",
       "\n",
       ".xr-attrs dt:hover span {\n",
       "  display: inline-block;\n",
       "  background: var(--xr-background-color);\n",
       "  padding-right: 10px;\n",
       "}\n",
       "\n",
       ".xr-attrs dd {\n",
       "  grid-column: 2;\n",
       "  white-space: pre-wrap;\n",
       "  word-break: break-all;\n",
       "}\n",
       "\n",
       ".xr-icon-database,\n",
       ".xr-icon-file-text2,\n",
       ".xr-no-icon {\n",
       "  display: inline-block;\n",
       "  vertical-align: middle;\n",
       "  width: 1em;\n",
       "  height: 1.5em !important;\n",
       "  stroke-width: 0;\n",
       "  stroke: currentColor;\n",
       "  fill: currentColor;\n",
       "}\n",
       "</style><pre class='xr-text-repr-fallback'>&lt;xarray.Dataset&gt;\n",
       "Dimensions:      (iterations: 2030, input_dim: 100, realization: 10,\n",
       "                  optimizer: 5, output_dim: 1)\n",
       "Coordinates:\n",
       "  * iterations   (iterations) int32 0 1 2 3 4 5 ... 2025 2026 2027 2028 2029\n",
       "  * input_dim    (input_dim) object &#x27;x0&#x27; &#x27;x1&#x27; &#x27;x10&#x27; &#x27;x11&#x27; ... &#x27;x97&#x27; &#x27;x98&#x27; &#x27;x99&#x27;\n",
       "  * realization  (realization) int32 0 1 2 3 4 5 6 7 8 9\n",
       "  * optimizer    (optimizer) object &#x27;CMAES&#x27; &#x27;PSO&#x27; &#x27;Adam&#x27; &#x27;LBFGSB&#x27; &#x27;RandomSearch&#x27;\n",
       "  * output_dim   (output_dim) object &#x27;y&#x27;\n",
       "Data variables:\n",
       "    input        (optimizer, realization, iterations, input_dim) float64 ...\n",
       "    output       (optimizer, realization, iterations, output_dim) float64 ...\n",
       "Attributes:\n",
       "    number_of_samples:        30\n",
       "    realization_seeds:        [874591 874592 874593 874594 874595 874596 8745...\n",
       "    function_seed:            874591\n",
       "    function_name:            StyblinskiTang\n",
       "    function_noise:           0.1\n",
       "    function_dimensionality:  100\n",
       "    function_global_minimum:  -3702.0910707861212\n",
       "    function_features:        noiseless,high_dimensional,seperable,differenti...</pre><div class='xr-wrap' style='display:none'><div class='xr-header'><div class='xr-obj-type'>xarray.Dataset</div></div><ul class='xr-sections'><li class='xr-section-item'><input id='section-6159fc58-ebd2-4f57-a938-c254d67d82a4' class='xr-section-summary-in' type='checkbox' disabled ><label for='section-6159fc58-ebd2-4f57-a938-c254d67d82a4' class='xr-section-summary'  title='Expand/collapse section'>Dimensions:</label><div class='xr-section-inline-details'><ul class='xr-dim-list'><li><span class='xr-has-index'>iterations</span>: 2030</li><li><span class='xr-has-index'>input_dim</span>: 100</li><li><span class='xr-has-index'>realization</span>: 10</li><li><span class='xr-has-index'>optimizer</span>: 5</li><li><span class='xr-has-index'>output_dim</span>: 1</li></ul></div><div class='xr-section-details'></div></li><li class='xr-section-item'><input id='section-a610aff2-d4d0-4e3a-b401-a9a076e3346f' class='xr-section-summary-in' type='checkbox'  checked><label for='section-a610aff2-d4d0-4e3a-b401-a9a076e3346f' class='xr-section-summary' >Coordinates: <span>(5)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>iterations</span></div><div class='xr-var-dims'>(iterations)</div><div class='xr-var-dtype'>int32</div><div class='xr-var-preview xr-preview'>0 1 2 3 4 ... 2026 2027 2028 2029</div><input id='attrs-c610f362-9310-4bcc-a0da-78577ba3b54c' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-c610f362-9310-4bcc-a0da-78577ba3b54c' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-81d30bb7-025f-47b9-9026-4583d59800d5' class='xr-var-data-in' type='checkbox'><label for='data-81d30bb7-025f-47b9-9026-4583d59800d5' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>array([   0,    1,    2, ..., 2027, 2028, 2029], dtype=int32)</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>input_dim</span></div><div class='xr-var-dims'>(input_dim)</div><div class='xr-var-dtype'>object</div><div class='xr-var-preview xr-preview'>&#x27;x0&#x27; &#x27;x1&#x27; &#x27;x10&#x27; ... &#x27;x98&#x27; &#x27;x99&#x27;</div><input id='attrs-86d81c3b-cd25-4dae-acdd-27d20fa5b559' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-86d81c3b-cd25-4dae-acdd-27d20fa5b559' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-82c4da4e-9ea3-4e47-90c2-588cfce98c44' class='xr-var-data-in' type='checkbox'><label for='data-82c4da4e-9ea3-4e47-90c2-588cfce98c44' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>array([&#x27;x0&#x27;, &#x27;x1&#x27;, &#x27;x10&#x27;, &#x27;x11&#x27;, &#x27;x12&#x27;, &#x27;x13&#x27;, &#x27;x14&#x27;, &#x27;x15&#x27;, &#x27;x16&#x27;, &#x27;x17&#x27;,\n",
       "       &#x27;x18&#x27;, &#x27;x19&#x27;, &#x27;x2&#x27;, &#x27;x20&#x27;, &#x27;x21&#x27;, &#x27;x22&#x27;, &#x27;x23&#x27;, &#x27;x24&#x27;, &#x27;x25&#x27;, &#x27;x26&#x27;,\n",
       "       &#x27;x27&#x27;, &#x27;x28&#x27;, &#x27;x29&#x27;, &#x27;x3&#x27;, &#x27;x30&#x27;, &#x27;x31&#x27;, &#x27;x32&#x27;, &#x27;x33&#x27;, &#x27;x34&#x27;, &#x27;x35&#x27;,\n",
       "       &#x27;x36&#x27;, &#x27;x37&#x27;, &#x27;x38&#x27;, &#x27;x39&#x27;, &#x27;x4&#x27;, &#x27;x40&#x27;, &#x27;x41&#x27;, &#x27;x42&#x27;, &#x27;x43&#x27;, &#x27;x44&#x27;,\n",
       "       &#x27;x45&#x27;, &#x27;x46&#x27;, &#x27;x47&#x27;, &#x27;x48&#x27;, &#x27;x49&#x27;, &#x27;x5&#x27;, &#x27;x50&#x27;, &#x27;x51&#x27;, &#x27;x52&#x27;, &#x27;x53&#x27;,\n",
       "       &#x27;x54&#x27;, &#x27;x55&#x27;, &#x27;x56&#x27;, &#x27;x57&#x27;, &#x27;x58&#x27;, &#x27;x59&#x27;, &#x27;x6&#x27;, &#x27;x60&#x27;, &#x27;x61&#x27;, &#x27;x62&#x27;,\n",
       "       &#x27;x63&#x27;, &#x27;x64&#x27;, &#x27;x65&#x27;, &#x27;x66&#x27;, &#x27;x67&#x27;, &#x27;x68&#x27;, &#x27;x69&#x27;, &#x27;x7&#x27;, &#x27;x70&#x27;, &#x27;x71&#x27;,\n",
       "       &#x27;x72&#x27;, &#x27;x73&#x27;, &#x27;x74&#x27;, &#x27;x75&#x27;, &#x27;x76&#x27;, &#x27;x77&#x27;, &#x27;x78&#x27;, &#x27;x79&#x27;, &#x27;x8&#x27;, &#x27;x80&#x27;,\n",
       "       &#x27;x81&#x27;, &#x27;x82&#x27;, &#x27;x83&#x27;, &#x27;x84&#x27;, &#x27;x85&#x27;, &#x27;x86&#x27;, &#x27;x87&#x27;, &#x27;x88&#x27;, &#x27;x89&#x27;, &#x27;x9&#x27;,\n",
       "       &#x27;x90&#x27;, &#x27;x91&#x27;, &#x27;x92&#x27;, &#x27;x93&#x27;, &#x27;x94&#x27;, &#x27;x95&#x27;, &#x27;x96&#x27;, &#x27;x97&#x27;, &#x27;x98&#x27;, &#x27;x99&#x27;],\n",
       "      dtype=object)</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>realization</span></div><div class='xr-var-dims'>(realization)</div><div class='xr-var-dtype'>int32</div><div class='xr-var-preview xr-preview'>0 1 2 3 4 5 6 7 8 9</div><input id='attrs-c4ea498e-38f6-4ab4-9ba8-54bb695c1907' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-c4ea498e-38f6-4ab4-9ba8-54bb695c1907' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-57c31ec6-29f6-4321-bcb5-a3b7d8ad2434' class='xr-var-data-in' type='checkbox'><label for='data-57c31ec6-29f6-4321-bcb5-a3b7d8ad2434' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int32)</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>optimizer</span></div><div class='xr-var-dims'>(optimizer)</div><div class='xr-var-dtype'>object</div><div class='xr-var-preview xr-preview'>&#x27;CMAES&#x27; &#x27;PSO&#x27; ... &#x27;RandomSearch&#x27;</div><input id='attrs-91099acb-b248-471e-ac5c-0cb617c9197e' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-91099acb-b248-471e-ac5c-0cb617c9197e' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-0865a370-2e3a-4d5e-b674-93d5c764c4b5' class='xr-var-data-in' type='checkbox'><label for='data-0865a370-2e3a-4d5e-b674-93d5c764c4b5' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>array([&#x27;CMAES&#x27;, &#x27;PSO&#x27;, &#x27;Adam&#x27;, &#x27;LBFGSB&#x27;, &#x27;RandomSearch&#x27;], dtype=object)</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span class='xr-has-index'>output_dim</span></div><div class='xr-var-dims'>(output_dim)</div><div class='xr-var-dtype'>object</div><div class='xr-var-preview xr-preview'>&#x27;y&#x27;</div><input id='attrs-58454e9b-c35d-4741-a99c-f949e3aff327' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-58454e9b-c35d-4741-a99c-f949e3aff327' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-9dbbb3a3-f36e-466f-8a68-347a3e7b7974' class='xr-var-data-in' type='checkbox'><label for='data-9dbbb3a3-f36e-466f-8a68-347a3e7b7974' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>array([&#x27;y&#x27;], dtype=object)</pre></div></li></ul></div></li><li class='xr-section-item'><input id='section-13a25fa8-18b4-4941-9f35-d2c607f66072' class='xr-section-summary-in' type='checkbox'  checked><label for='section-13a25fa8-18b4-4941-9f35-d2c607f66072' class='xr-section-summary' >Data variables: <span>(2)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'><li class='xr-var-item'><div class='xr-var-name'><span>input</span></div><div class='xr-var-dims'>(optimizer, realization, iterations, input_dim)</div><div class='xr-var-dtype'>float64</div><div class='xr-var-preview xr-preview'>...</div><input id='attrs-f55a15e9-f51c-482e-8777-9aacaa776acb' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-f55a15e9-f51c-482e-8777-9aacaa776acb' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-5b23c1c8-9d82-47a0-8f81-2d586b44af72' class='xr-var-data-in' type='checkbox'><label for='data-5b23c1c8-9d82-47a0-8f81-2d586b44af72' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>[10150000 values with dtype=float64]</pre></div></li><li class='xr-var-item'><div class='xr-var-name'><span>output</span></div><div class='xr-var-dims'>(optimizer, realization, iterations, output_dim)</div><div class='xr-var-dtype'>float64</div><div class='xr-var-preview xr-preview'>...</div><input id='attrs-a9c66ec7-5db3-4fce-9078-4645e7d9b490' class='xr-var-attrs-in' type='checkbox' disabled><label for='attrs-a9c66ec7-5db3-4fce-9078-4645e7d9b490' title='Show/Hide attributes'><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg></label><input id='data-42928777-da71-4ae0-bf6d-a1d5a237f3ef' class='xr-var-data-in' type='checkbox'><label for='data-42928777-da71-4ae0-bf6d-a1d5a237f3ef' title='Show/Hide data repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-var-attrs'><dl class='xr-attrs'></dl></div><div class='xr-var-data'><pre>[101500 values with dtype=float64]</pre></div></li></ul></div></li><li class='xr-section-item'><input id='section-80b2ee53-283d-4985-867f-25b68da8eddb' class='xr-section-summary-in' type='checkbox'  ><label for='section-80b2ee53-283d-4985-867f-25b68da8eddb' class='xr-section-summary' >Indexes: <span>(5)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><ul class='xr-var-list'><li class='xr-var-item'><div class='xr-index-name'><div>iterations</div></div><div class='xr-index-preview'>PandasIndex</div><div></div><input id='index-5dd33694-5ffa-4216-add5-f4be8810dbee' class='xr-index-data-in' type='checkbox'/><label for='index-5dd33694-5ffa-4216-add5-f4be8810dbee' title='Show/Hide index repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-index-data'><pre>PandasIndex(Index([   0,    1,    2,    3,    4,    5,    6,    7,    8,    9,\n",
       "       ...\n",
       "       2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029],\n",
       "      dtype=&#x27;int32&#x27;, name=&#x27;iterations&#x27;, length=2030))</pre></div></li><li class='xr-var-item'><div class='xr-index-name'><div>input_dim</div></div><div class='xr-index-preview'>PandasIndex</div><div></div><input id='index-a890de14-5f1e-4246-93e7-72830ffa3c19' class='xr-index-data-in' type='checkbox'/><label for='index-a890de14-5f1e-4246-93e7-72830ffa3c19' title='Show/Hide index repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-index-data'><pre>PandasIndex(Index([&#x27;x0&#x27;, &#x27;x1&#x27;, &#x27;x10&#x27;, &#x27;x11&#x27;, &#x27;x12&#x27;, &#x27;x13&#x27;, &#x27;x14&#x27;, &#x27;x15&#x27;, &#x27;x16&#x27;, &#x27;x17&#x27;,\n",
       "       &#x27;x18&#x27;, &#x27;x19&#x27;, &#x27;x2&#x27;, &#x27;x20&#x27;, &#x27;x21&#x27;, &#x27;x22&#x27;, &#x27;x23&#x27;, &#x27;x24&#x27;, &#x27;x25&#x27;, &#x27;x26&#x27;,\n",
       "       &#x27;x27&#x27;, &#x27;x28&#x27;, &#x27;x29&#x27;, &#x27;x3&#x27;, &#x27;x30&#x27;, &#x27;x31&#x27;, &#x27;x32&#x27;, &#x27;x33&#x27;, &#x27;x34&#x27;, &#x27;x35&#x27;,\n",
       "       &#x27;x36&#x27;, &#x27;x37&#x27;, &#x27;x38&#x27;, &#x27;x39&#x27;, &#x27;x4&#x27;, &#x27;x40&#x27;, &#x27;x41&#x27;, &#x27;x42&#x27;, &#x27;x43&#x27;, &#x27;x44&#x27;,\n",
       "       &#x27;x45&#x27;, &#x27;x46&#x27;, &#x27;x47&#x27;, &#x27;x48&#x27;, &#x27;x49&#x27;, &#x27;x5&#x27;, &#x27;x50&#x27;, &#x27;x51&#x27;, &#x27;x52&#x27;, &#x27;x53&#x27;,\n",
       "       &#x27;x54&#x27;, &#x27;x55&#x27;, &#x27;x56&#x27;, &#x27;x57&#x27;, &#x27;x58&#x27;, &#x27;x59&#x27;, &#x27;x6&#x27;, &#x27;x60&#x27;, &#x27;x61&#x27;, &#x27;x62&#x27;,\n",
       "       &#x27;x63&#x27;, &#x27;x64&#x27;, &#x27;x65&#x27;, &#x27;x66&#x27;, &#x27;x67&#x27;, &#x27;x68&#x27;, &#x27;x69&#x27;, &#x27;x7&#x27;, &#x27;x70&#x27;, &#x27;x71&#x27;,\n",
       "       &#x27;x72&#x27;, &#x27;x73&#x27;, &#x27;x74&#x27;, &#x27;x75&#x27;, &#x27;x76&#x27;, &#x27;x77&#x27;, &#x27;x78&#x27;, &#x27;x79&#x27;, &#x27;x8&#x27;, &#x27;x80&#x27;,\n",
       "       &#x27;x81&#x27;, &#x27;x82&#x27;, &#x27;x83&#x27;, &#x27;x84&#x27;, &#x27;x85&#x27;, &#x27;x86&#x27;, &#x27;x87&#x27;, &#x27;x88&#x27;, &#x27;x89&#x27;, &#x27;x9&#x27;,\n",
       "       &#x27;x90&#x27;, &#x27;x91&#x27;, &#x27;x92&#x27;, &#x27;x93&#x27;, &#x27;x94&#x27;, &#x27;x95&#x27;, &#x27;x96&#x27;, &#x27;x97&#x27;, &#x27;x98&#x27;, &#x27;x99&#x27;],\n",
       "      dtype=&#x27;object&#x27;, name=&#x27;input_dim&#x27;))</pre></div></li><li class='xr-var-item'><div class='xr-index-name'><div>realization</div></div><div class='xr-index-preview'>PandasIndex</div><div></div><input id='index-d2933c08-d97c-45e7-8b22-ab8101615120' class='xr-index-data-in' type='checkbox'/><label for='index-d2933c08-d97c-45e7-8b22-ab8101615120' title='Show/Hide index repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-index-data'><pre>PandasIndex(Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=&#x27;int32&#x27;, name=&#x27;realization&#x27;))</pre></div></li><li class='xr-var-item'><div class='xr-index-name'><div>optimizer</div></div><div class='xr-index-preview'>PandasIndex</div><div></div><input id='index-fa8babc9-898d-4496-819b-02c28f4fa37e' class='xr-index-data-in' type='checkbox'/><label for='index-fa8babc9-898d-4496-819b-02c28f4fa37e' title='Show/Hide index repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-index-data'><pre>PandasIndex(Index([&#x27;CMAES&#x27;, &#x27;PSO&#x27;, &#x27;Adam&#x27;, &#x27;LBFGSB&#x27;, &#x27;RandomSearch&#x27;], dtype=&#x27;object&#x27;, name=&#x27;optimizer&#x27;))</pre></div></li><li class='xr-var-item'><div class='xr-index-name'><div>output_dim</div></div><div class='xr-index-preview'>PandasIndex</div><div></div><input id='index-9ace96bc-a80b-45fd-9ba3-f1b2c50023de' class='xr-index-data-in' type='checkbox'/><label for='index-9ace96bc-a80b-45fd-9ba3-f1b2c50023de' title='Show/Hide index repr'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg></label><div class='xr-index-data'><pre>PandasIndex(Index([&#x27;y&#x27;], dtype=&#x27;object&#x27;, name=&#x27;output_dim&#x27;))</pre></div></li></ul></div></li><li class='xr-section-item'><input id='section-5a3b4319-d7ee-4c75-9cff-4fb0e2521fdf' class='xr-section-summary-in' type='checkbox'  checked><label for='section-5a3b4319-d7ee-4c75-9cff-4fb0e2521fdf' class='xr-section-summary' >Attributes: <span>(8)</span></label><div class='xr-section-inline-details'></div><div class='xr-section-details'><dl class='xr-attrs'><dt><span>number_of_samples :</span></dt><dd>30</dd><dt><span>realization_seeds :</span></dt><dd>[874591 874592 874593 874594 874595 874596 874597 874598 874599 874600]</dd><dt><span>function_seed :</span></dt><dd>874591</dd><dt><span>function_name :</span></dt><dd>StyblinskiTang</dd><dt><span>function_noise :</span></dt><dd>0.1</dd><dt><span>function_dimensionality :</span></dt><dd>100</dd><dt><span>function_global_minimum :</span></dt><dd>-3702.0910707861212</dd><dt><span>function_features :</span></dt><dd>noiseless,high_dimensional,seperable,differentiable,multimodal</dd></dl></div></li></ul></div></div>"
      ],
      "text/plain": [
       "<xarray.Dataset>\n",
       "Dimensions:      (iterations: 2030, input_dim: 100, realization: 10,\n",
       "                  optimizer: 5, output_dim: 1)\n",
       "Coordinates:\n",
       "  * iterations   (iterations) int32 0 1 2 3 4 5 ... 2025 2026 2027 2028 2029\n",
       "  * input_dim    (input_dim) object 'x0' 'x1' 'x10' 'x11' ... 'x97' 'x98' 'x99'\n",
       "  * realization  (realization) int32 0 1 2 3 4 5 6 7 8 9\n",
       "  * optimizer    (optimizer) object 'CMAES' 'PSO' 'Adam' 'LBFGSB' 'RandomSearch'\n",
       "  * output_dim   (output_dim) object 'y'\n",
       "Data variables:\n",
       "    input        (optimizer, realization, iterations, input_dim) float64 ...\n",
       "    output       (optimizer, realization, iterations, output_dim) float64 ...\n",
       "Attributes:\n",
       "    number_of_samples:        30\n",
       "    realization_seeds:        [874591 874592 874593 874594 874595 874596 8745...\n",
       "    function_seed:            874591\n",
       "    function_name:            StyblinskiTang\n",
       "    function_noise:           0.1\n",
       "    function_dimensionality:  100\n",
       "    function_global_minimum:  -3702.0910707861212\n",
       "    function_features:        noiseless,high_dimensional,seperable,differenti..."
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Go out of presentation mode to see this cell in its entirety\n",
    "\n",
    "l2o.open_one_dataset_raw(small_dataset, 280)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "936252c8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Aim of the project\n",
    "\n",
    "Design a general *data-driven optimization strategy* with machine learning that competes with classic hand-engineered optimizers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26bff0b3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You will have to fill in the template code provided in the `l2o.py` file:\n",
    "* Implement the `predict()` function of your custom strategy!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "119b618a",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "class MyStrategy(l2o.CustomStrategy):\n",
    "    name: str = \"custom_strategy\"\n",
    "\n",
    "    def predict(self, features):\n",
    "        ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2073d0ce",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "<img src=\"./img/predict.png\" width=\"35%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaaef2f2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The project consists of answering 4 questions \n",
    "\n",
    "* The first 3 questions let you explore the datasets and teaches you how to train a basic classification model.  \n",
    "* In the last question you can show your creativity in fitting machine learning model of your choice! "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c289eb9",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## [Black-Box Optimization Benchmarking BBOB](https://numbbo.github.io/workshops/BBOB-2022/index.html) competition"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21a77245",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=./img/Brown_logo.svg width=30%>\n",
    "\n",
    "# Project 1: Learning to Choose Optimizers\n",
    "\n",
    "\n",
    "### Martin van der Schelling | <a href = \"mailto: martin_van_der_schelling@brown.edu\">martin_van_der_schelling@brown.edu</a>  | PhD candidate"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:presentation] *",
   "language": "python",
   "name": "conda-env-presentation-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}