docs/source/Quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quickstart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we will show how to solve a famous optimization problem, minimizing the Rosenbrock function, in simplenlopt. First, let's define the Rosenbrock function and plot it:\n",
    "\n",
    "$$\n",
    "f(x, y) = (1-x)^2+100(y-x^2)^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def rosenbrock(pos):\n",
    "    \n",
    "    x, y = pos\n",
    "    return (1-x)**2 + 100 * (y - x**2)**2\n",
    "\n",
    "xgrid = np.linspace(-2, 2, 500)\n",
    "ygrid = np.linspace(-1, 3, 500)\n",
    "\n",
    "X, Y = np.meshgrid(xgrid, ygrid)\n",
    "\n",
    "Z = (1 - X)**2 + 100 * (Y -X**2)**2\n",
    "\n",
    "x0=np.array([-1.5, 2.25])\n",
    "f0 = rosenbrock(x0)\n",
    "\n",
    "#Plotly not rendering correctly on Readthedocs, but this shows how it is generated! Plot below is a PNG export\n",
    "import plotly.graph_objects as go\n",
    "fig = go.Figure(data=[go.Surface(z=Z, x=X, y=Y, cmax = 10, cmin = 0, showscale = False)])\n",
    "fig.update_layout(\n",
    "    scene = dict(zaxis = dict(nticks=4, range=[0,10])))\n",
    "fig.add_scatter3d(x=[1], y=[1], z=[0], mode = 'markers', marker=dict(size=10, color='green'), name='Optimum')\n",
    "fig.add_scatter3d(x=[-1.5], y=[2.25], z=[f0], mode = 'markers', marker=dict(size=10, color='black'), name='Initial guess')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![The Rosenbrock optimization problem](./Rosenbrock.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The crux of the Rosenbrock function is that the minimum indicated by the green dot is located in a very narrow, banana shaped valley with a small slope around the minimum. Local optimizers try to find the optimum by searching the parameter space starting from an initial guess. We place the initial guess shown in black on the other side of the banana. \n",
    "\n",
    "In simplenlopt, local optimizers are called by the minimize function. It requires the objective function and a starting point. The algorithm is chosen by the method argument. Here, we will use the derivative-free Nelder-Mead algorithm. Objective functions must be of the form ``f(x, ...)`` where ``x`` represents a numpy array holding the parameters which are optimized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Position of optimum:  [0.99999939 0.9999988 ]\n",
      "Function value at Optimum:  4.0813291938703456e-13\n",
      "Number of function evaluations:  232\n"
     ]
    }
   ],
   "source": [
    "import simplenlopt\n",
    "\n",
    "def rosenbrock(pos):\n",
    "    \n",
    "    x, y = pos\n",
    "    return (1-x)**2 + 100 * (y - x**2)**2\n",
    "\n",
    "res = simplenlopt.minimize(rosenbrock, x0, method = 'neldermead')\n",
    "print(\"Position of optimum: \", res.x)\n",
    "print(\"Function value at Optimum: \", res.fun)\n",
    "print(\"Number of function evaluations: \", res.nfev)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The optimization result is stored in a class whose main attributes are the position of the optimum and the function value at the optimum. The number of function evaluations is a measure of performance: the less function evaluations are required to find the minimum, the faster the optimization will be.\n",
    "\n",
    "Next, let's switch to a derivative based solver. For better performance, we also supply the analytical gradient which is passed to the jac argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Position of optimum:  [1. 1.]\n",
      "Function value at Optimum:  1.53954903146237e-20\n",
      "Number of function evaluations:  75\n"
     ]
    }
   ],
   "source": [
    "def rosenbrock_grad(pos):\n",
    "    \n",
    "    x, y = pos\n",
    "    dx = 2 * x -2 - 400 * x * (y-x**2)\n",
    "    dy = 200 * (y-x**2)\n",
    "    \n",
    "    return dx, dy\n",
    "\n",
    "res_slsqp = simplenlopt.minimize(rosenbrock, x0, method = 'slsqp', jac = rosenbrock_grad)\n",
    "print(\"Position of optimum: \", res_slsqp.x)\n",
    "print(\"Function value at Optimum: \", res_slsqp.fun)\n",
    "print(\"Number of function evaluations: \", res_slsqp.nfev)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the SLSQP algorithm can use gradient information, it requires less function evaluations to find the minimum than the \n",
    "derivative-free Nelder-Mead algorithm. \n",
    "\n",
    "Unlike vanilla NLopt, simplenlopt automatically defaults to finite difference approximations of the gradient if it is \n",
    "not provided:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Position of optimum:  [0.99999999 0.99999999]\n",
      "Function value at Optimum:  5.553224195710645e-17\n",
      "Number of function evaluations:  75\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\danie\\.conda\\envs\\simplenlopt_env\\lib\\site-packages\\simplenlopt\\_Core.py:178: RuntimeWarning: Using gradient-based optimization, but no gradient information is available. Gradient will be approximated by central difference. Consider using a derivative-free optimizer or supplying gradient information.\n",
      "  warn('Using gradient-based optimization'\n"
     ]
    }
   ],
   "source": [
    "res = simplenlopt.minimize(rosenbrock, x0, method = 'slsqp')\n",
    "print(\"Position of optimum: \", res.x)\n",
    "print(\"Function value at Optimum: \", res.fun)\n",
    "print(\"Number of function evaluations: \", res.nfev)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the finite differences are not as precise as the analytical gradient, the found optimal function value is higher than with analytical gradient information. In general, it is aways recommended to compute the gradient analytically or by automatic differentiation as the inaccuracies of finite differences can result in wrong results and bad performance.\n",
    "\n",
    "For demonstration purposes, let's finally solve the problem with a global optimizer. Like in SciPy, each global optimizer is called by a dedicated function such as crs() for the Controlled Random Search algorithm. Instead of a starting point, the global optimizers require a region in which they seek to find the minimum. This region is provided as a list of (lower_bound, upper_bound) for each coordinate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Position of optimum:  [1.00000198 1.00008434]\n",
      "Function value at Optimum:  6.45980535135501e-07\n",
      "Number of function evaluations:  907\n"
     ]
    }
   ],
   "source": [
    "bounds = [(-2., 2.), (-2., 2.)]\n",
    "res_crs = simplenlopt.crs(rosenbrock, bounds)\n",
    "print(\"Position of optimum: \", res_crs.x)\n",
    "print(\"Function value at Optimum: \", res_crs.fun)\n",
    "print(\"Number of function evaluations: \", res_crs.nfev)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that using a global optimizer is overkill for a small problem like the Rosenbrock function: it requires many more function\n",
    "evaluations than a local optimizer. Global optimization algorithms shine in case of complex, multimodal functions where local\n",
    "optimizers converge to local minima instead of the global minimum. Check the Global Optimization page for such an example. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}