6a.Optimization-Intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Finding the minimum of a smooth function\n",
    "\n",
    "Demos various methods to find the minimum of a function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SciPy optimize provides functions for minimizing (or maximizing) objective functions, possibly subject to constraints. It includes solvers for nonlinear problems (with support for both local and global optimization algorithms), linear programing, constrained and nonlinear least-squares, root finding and curve fitting.\n",
    "\n",
    "1. Scalar Functions Optimization\n",
    "2. Local (Multivariate) Optimization\n",
    "3. Global Optimization\n",
    "4. Least-squares and Curve Fitting\n",
    "5. Root finding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x29eab0a0e80>]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XlYVnX+//Hnmx0EQVYRcMcFzRW3LEtN28uaFm1qbJ+tmWqmvtNMzUwzTTPTLDXTr5kay8rKzPZF29TMVhfccAEFQQFFQAUUkPX+/P647+ZiDBS5l3Mv78d1eQH3feC8ONy+OHzOOZ8jxhiUUkr5ryCrAyillHIvLXqllPJzWvRKKeXntOiVUsrPadErpZSf06JXSik/p0WvlFJ+ToteKaX8nBa9Ukr5uRCrAwAkJiaa/v37Wx1DKaV8ysaNGw8ZY5JOtZxXFH3//v3JycmxOoZSSvkUEdnXleV06EYppfycFr1SSvk5LXqllPJzWvRKKeXntOiVUsrPadErpZSf06JXSik/59NFv6mkmkc+zLc6hlJKeTWfLvrt+2t58tM9FFbWWR1FKaW8lk8X/XnDUwD4eOdBi5MopZT38umi7xMXyci0nqzYWWF1FKWU8lo+XfQAs7N6s6W0hspjjVZHUUopr+TzRT8rKwVjYFVepdVRlFLKK/l80Q/rHUN6r0gdvlFKqU6csuhF5FkRqRSR7e0eixeRFSJS4Hjby/G4iMjjIlIoIrkiMs6d4R3rZFZWCl8UHqK+qdXdq1NKKZ/TlT3654ELTnjsPmCVMSYTWOX4GOBCINPx73bgSdfEPLlZWSk0t9r4vKDKE6tTSimfcsqiN8Z8Bhw54eHLgUWO9xcBc9o9/oKxWwvEiUiqq8J2ZmL/eGIjQ/lYh2+UUupbujtGn2KMKQdwvE12PJ4GlLZbrszxmFuFBAcxY1gyn+RX0tpmc/fqlFLKp7j6YKx08JjpcEGR20UkR0RyqqqcH3KZlZVCTUMLOfuqnf5aSinlT7pb9BXfDMk43n5zbmMZkNFuuXTgQEdfwBizwBiTbYzJTko65b1tT2nakCTCgoP07BullDpBd4v+XWC+4/35wDvtHv+e4+ybyUDtN0M87hYdHsKZgxNYsbMCYzr8I0IppQJSV06vXAJ8DQwVkTIRuQX4MzBLRAqAWY6PAd4HioBC4GngR25J3YlZWSmUHGlgV8UxT65WKaW8WsipFjDGzOvkqZkdLGuAHzsbqrvOG57C/W9tZ8WOCob17mlVDKWU8io+f2Vseyk9IxidEceKPB2nV0qpb/hV0QPMzkoht6yWg7U6yZlSSoGfFj2ge/VKKeXgd0U/ODma/glRepqlUko5+F3RfzPJ2dd7DnGsscXqOEopZTm/K3qAWVm9aWkzrNmtk5wppbxTS5uNe1/bSoEHTgf3y6If368X8T3CdPhGKeWVjDH89t0dvLaxjB0Hjrp9fX5Z9MFBwoxhyazOr6RFJzlTSnmZRV/t5eV1JfzgnEHMGev2eR/9s+jBfpXs0cZW1hefOMOyUkpZ54Nt5fxu2U5mZaVw7/lDPbJOvy36szMTCQ/RSc6UUt5jffER7ly6hbEZcTw+dyzBQR1N+Ot6flv0UWEhnJ2ZqJOcKaW8wu6KY9y6aAMZvSJZOH8CkWHBHlu33xY92Idv9tcc98jBDqWU6kxhZR3ffWYdEaHBLLp5Ir16hHl0/X5d9DOGpSCCDt8opSyzp6qOeU+vxRh4+bZJpPeK8ngGvy76pJhwxvXtpUWvlLJEUVUd8xasxRjDktsmMTg5xpIcfl30YJ/7Zmf5UcqqG6yOopQKIMWH6pn39FrabIaXb5tMZoo1JQ8BUPSzHJOcrdS9eqWUh+w9VM+8BWtpabOX/BALSx4CoOgHJkUzKKmHzmaplPKIfYfte/JNrW28fNskhva2tuQhAIoe7HPfrCs6Qu1xneRMKeU+JYcbmLdgLY0tbbx822SvudNdgBR9Cq02w6e7Kq2OopTyU6VHGpi74GsaWtpYfOtkhqd6R8lDgBT92Iw4EqPD+VjH6ZVSbnCg5jjznl5LfXMbi2+dRFYf7yl5CJCiDwoSzhuezJpdVTS1tlkdRynlRyqONnLd02upbWjhxVsmMqJPrNWRviUgih7swzd1Ta2sLdJJzpRSrnGoronrnl5L1bEmnr95IqPS46yO1KGAKfqpgxOJDA1mxc6DVkdRSvmB6vpmrn9mHQdqGnn2xgmM79fL6kidCpiijwgNZtoQ+yRnNptOcqaU6r66plZueHYdRYfqeWZ+NpMGJlgd6aQCpujBfpplxdEmtu2vtTqKUspHtbTZ+NHiTeSVH+Op68cxdXCi1ZFOKaCKfsawZIJ0kjOlVDcZY/jlm9v4bHcVf7xiJDOGpVgdqUucKnoRuVtEdojIdhFZIiIRIjJARNaJSIGILBURz87HeRLxPcLI7h+vRa+U6pbHVhbw+sYy7pyZybUT+lodp8u6XfQikgb8FMg2xowEgoG5wCPAY8aYTKAauMUVQV1ldlYKuyqOUXJYJzlTSnXdstwDPL6qgKvHp3PXeZlWxzktzg7dhACRIhICRAHlwAzgdcfzi4A5Tq7DpWZn9QbgYz37RinVRfkHj3Lva7mM79eLh684AxHP3ALQVbpd9MaY/cDfgBLsBV8LbARqjDGtjsXKAPff4vw09E2IYmhKjA7fKKW6pKahmdtf2EhMRAhPfnccYSG+d2jTmaGbXsDlwACgD9ADuLCDRTs8l1FEbheRHBHJqaqq6m6MbpmVlcKGvUeorm/26HqVUr7FZjPc+coWymuP8+T140nuGWF1pG5x5lfTeUCxMabKGNMCvAmcCcQ5hnIA0oEDHX2yMWaBMSbbGJOdlJTkRIzTNysrBZuBT/J1kjOlVOee/ryINburePCyEV59QdSpOFP0JcBkEYkS+4DVTGAnsBq4yrHMfOAd5yK63hlpsaT0DNfhG6VUp7bvr+VvH+/iwpG9uW6i75xh0xFnxujXYT/ougnY5vhaC4BfAD8TkUIgAVjogpwuZZ/kLIXPCqpobNFJzpRS/6uhuZWfvrKZhB7h/OlK3zv4eiKnjioYY35rjBlmjBlpjLnBGNNkjCkyxkw0xgw2xlxtjGlyVVhXmpWVQkNzG1/tOWR1FKWUl3loWR7Fh+p59NrRxEV5zaVA3eZ7h49dZMqgBKLDQ3T4Rin1P9bsrmLJ+hJunzaQMwd5//QGXRGwRR8eEsw5Q5JYsbNSJzlTSgH2IZv739rGoKQe/GzWEKvjuEzAFj3Yh28O1TWxubTG6ihKKS/w6Me7Kas+zp+/M4rwkGCr47hMQBf99KHJBAeJDt8opcgtq+HZL4u5blJfJvSPtzqOSwV00cdGhTJ5YDwf7zyIMTp8o1Sgam2zcd8b20iMDue+C4dZHcflArroAS4YmUpRVT35B49ZHUUpZZHF60rYWX6U3102gp4RoVbHcbmAL/oLR/YmOEh4b2uHF/AqpfxcTUMzj63czZmDErhgZG+r47hFwBd9YnQ4Zw5KYFluuQ7fKBWA/rmqgKPHW/j1JVk+f2FUZwK+6AEuHdWHkiMNeotBpQJMYWUdL369j7kT+zI8tafVcdxGix44f0RvQoN1+EapQPPw8p1Ehgb71TnzHdGix372zdmZSSzPLdeLp5QKEF8WHmL1rip+MnMwidHhVsdxKy16h0tHp3KgtpHNpdVWR1FKuZkxhr9+tIs+sRHMP7O/1XHcTove4bzhKYSFBPHe1nKroyil3GxVXiVbSmv46cxMv7oCtjNa9A4xEaHMGJrM8m3ltLbZrI6jlHITm83w9xW76ZcQxXfGp1sdxyO06NuZM7YPVcea+HLPYaujKKXc5P3t5eSVH+Xu84YQGhwYFRgY32UXTR+WTGxkKG9uKrM6ilLKDVrbbDy6YjeZydFcOrqP1XE8Rou+nfCQYC4ZlcpHOw5S19RqdRyllIstyy2nqKqen80aQnCQf14c1REt+hNcOS6dxhYbH2zTg7JK+RObzfDvTwsZkhLN+SP8c6qDzmjRn2Bc3zj6J0Tx5qb9VkdRSrnQqvxKdlfU8aNzBxMUQHvzoEX/LSLClePS+broMGXVDVbHUUq5gDGGJ1YXkhEfySWjUq2O43Fa9B24YmwaAO9s0SkRlPIHX+85zNbSGr4/bRAhAXKmTXuB9x13QUZ8FBP7x/PGpjKd0VIpP/CvTwtJignnqgA5b/5EWvSduGp8OkVV9Wzcp1MiKOXLtpbW8GXhYW47ewARof5/FWxHtOg7ccnoVKLDQ3h5fYnVUZRSTnj68yJiwkO4blI/q6NYRou+E1FhIcwZ24flueXUNrRYHUcp1Q37a47zwfaDzJ2YQXR4iNVxLKNFfxLzJvalqdXGW5v1SlmlfNELX+0FCIgZKk/GqaIXkTgReV1E8kUkT0SmiEi8iKwQkQLH216uCutpI/rEMjo9liXrS/WgrFI+pr6plZfXl3DByN6k94qyOo6lnN2j/yfwoTFmGDAayAPuA1YZYzKBVY6Pfda8iX3ZVXGMTSU1VkdRSp2G1zeWcayxlVvOGmB1FMt1u+hFpCcwDVgIYIxpNsbUAJcDixyLLQLmOBvSSpeO7kOPsGCW6EFZpXyGzWZ47stixvaNY1xfnx1UcBln9ugHAlXAcyKyWUSeEZEeQIoxphzA8Ta5o08WkdtFJEdEcqqqqpyI4V49wkO4fGway3IPUNPQbHUcpVQXfJJfyd7DDbo37+BM0YcA44AnjTFjgXpOY5jGGLPAGJNtjMlOSkpyIob7fW9KPxpbbHqqpVI+4oW1++jdM4ILAmzyss44U/RlQJkxZp3j49exF3+FiKQCON5WOhfResN69+TszEQWfbWX5la9+5RS3mzvoXo+213FvIl9A3K6g450eysYYw4CpSIy1PHQTGAn8C4w3/HYfOAdpxJ6iZvPGkDF0SaWb9P5b5TyZi+vLyEkSJg7McPqKF7D2SsIfgIsFpEwoAi4Cfsvj1dF5BagBLjayXV4hXMykxicHM3CL4qZMyYNkcCa5lQpX9DY0sarOaXMHpFCSs8Iq+N4DaeK3hizBcju4KmZznxdbxQUJNw8dQC/emsb64qPMHlggtWRlFInWJ5bTk1DC9dPDtzpDjqiA1in4cpxafSKCmXhF8VWR1FKdeDFtfsYlNSDKboj9j+06E9DRGgwN0zux8q8Cgorj1kdRynVzvb9tWwpreH6yf10aPUEWvSn6capA4gMDebxVYVWR1FKtfPS2n1EhgZz5bjAnHP+ZLToT1N8jzDmn9mf93IPUFChe/VKeYPa4y28vWU/l4/pQ2xkqNVxvI4WfTfcdvZAokKDefwT3atXyhu8uamMxhabHoTthBZ9N3yzV78s9wC7da9eKUsZY3hp7T7GZMQxMi3W6jheSYu+m/67V7+qwOooSgW0DXur2VNVz3cn9bU6itfSou+mXj3CuHFqf5ZvK2fHgVqr4ygVsJZuKCU6PISLR6VaHcVradE74fZpg+gVFcbv3t2pNyZRygJHG1tYvu0Al43pQ1RY4N4q8FS06J0QGxnKPbOHsn7vEZblllsdR6mA8+6WAzS22Jg7Qee1ORkteiddOyGDEX168sf382hobrU6jlIBZemGUoan9uQMPQh7Ulr0TgoOEn576QjKaxt56tM9VsdRKmBs31/Ltv21zJ2QoVfCnoIWvQtMHBDPpaP78NRnRew7XG91HKUCwqs5pYSFBDFnTJrVUbyeFr2L/OqiYYQFB/GLN3Kx2fTArFLu1NjSxlub93PRyN7ERumVsKeiRe8iqbGR3H/xcNYWHWGx3nJQKbf6YHs5xxpbuUYPwnaJFr0LzZ2QwdmZifxxeR6FlXVWx1HKb72yvpR+CVFMHqDTEXeFFr0LiQh/u3o0kWHB/GTJZhpb2qyOpJTfKaqqY13xEa7JziAoSA/CdoUWvYul9Izgr1eNIq/8KA++u0MvpFLKxV7NKSM4SLhqvE5H3FVa9G4wc3gKd0wfzCsbSnlpnY7XK+UqLW02Xt9YxvShyXpP2NOgRe8md88awvShSTz47g5W51daHUcpv7A6v5JDdU16Jexp0qJ3k+Ag4f9dN47hqTH8aPEmNu6rtjqSUj5v6YZSkmPCOXdoktVRfIoWvRtFh4fw7I0TSOkZzo3PrmdziZa9Ut11sLaR1bsquWp8OiHBWl2nQ7eWmyXHRLDk9snER4dx/TPr+LygyupISvmk1zeWYjP2+aXU6dGi94DU2Ehe/f4UMuKjuOm5DSzRC6qUOi02m2FpTilTBibQL6GH1XF8jha9h6T0jODVH0xhyqAEfvnmNu59bSv1TTrbpVJdsbboMKVHjjN3ou7Nd4cWvQf1jAjl+Zsmcsf0wby+qYwL//m5DuUo1QWvbCilZ0QI54/obXUUn+R00YtIsIhsFpFljo8HiMg6ESkQkaUiEuZ8TP8RHCTcc/5QXrltMiJww8L13Pjcegr0JuNKdai6vpkPtx/kirFpRIQGWx3HJ7lij/5OIK/dx48AjxljMoFq4BYXrMPvTBqYwMd3T+NXFw1j475qLvjn59z5ymY27qvWq2mVauftLftpbrNx7QS9+Xd3OVX0IpIOXAw84/hYgBnA645FFgFznFmHPwsPCeb2aYNYc+90bjyzP6vyKvnOk19x2RNf8uLXe6k81mh1RKUsZYxh6YZSzkiLJatPT6vj+Cxn9+j/AfwfYHN8nADUGGO+OcpYBnR4VwARuV1EckQkp6oqsMep43uE8etLslj7q5k8NGckza02fv3ODib9cRXX/Odrnvm8iD1VdbqnrwJOblkt+QeP6SmVTur2bdNF5BKg0hizUUTO/ebhDhbtsJ2MMQuABQDZ2dnaYNgvsLphcj9umNyP3RXHeH9bOR9sO8gflufxh+V59I2PYvrQJKYPS2bywAQdr1R+75UNpUSEBnHZmD5WR/Fp3S56YCpwmYhcBEQAPbHv4ceJSIhjrz4dOOB8zMAzJCWGISkx3HXeEEqPNPDp7io+za9kaU4pi77eR0RoEGcOSmT60CTOHZpMRnyU1ZGVcqmG5lbe23qAi85IpWeE3kXKGd0uemPML4FfAjj26O8xxnxXRF4DrgJeAeYD77ggZ0DLiI/6755+Y0sba4sO8+muKj7Jr+ST/EpgB1mpPbl2QgZzxqYRG6n/KZTvW55bTl1TK3P1IKzTxBXjvu2K/hIRGYi95OOBzcD1xpimk31+dna2ycnJcTpHoDHGUHyonk/yK3l7y3627z9KeEgQl4/pwx3TM+mboHv5yndd9eRXHKlvZtXPz8F+noc6kYhsNMZkn2o5Z4Zu/ssY8ynwqeP9ImCiK76uOjkRYWBSNAOTorn17IFs31/LkvUlvL6xjDc37eeq8encMWMw6b208JVvKaysI2dfNfddOExL3gX0ylg/MjItloevOIPP/m8610/ux5ub9nPeo2t4+rMi2mx6vFv5jldzSgkJEq4c1+FJe+o0adH7oZSeETx42QhW33suZw1O4uH387jy31+y66Befau8X3OrjTc2ljFzeDLJMXoXKVfQovdjaXGRPP298Tw+byyl1ce59IkveGtzmdWxlDqpFTsrOFzfrOfOu5AWvZ8TES4b3YcVd09jXN847l66lT++n6dDOcprvbR2H2lxkZwzJNnqKH5Diz5AJESH8+Itk7hhcj8WfFbELYs2cLy5zepYSv2PwspjfF10mOsm9SU4SA/CuooWfQAJDQ7ioTkjefiKkXy2u4qbn99AQ7POia+8x0trSwgNFh22cTEt+gD03Un9ePSaMawrPqxlr7xGQ3Mrb2wq48KRqSRGh1sdx69o0QeoOWPTeOzaMawvPsJNz22gsUWHcZS13t1ygGONrdwwpZ/VUfyOFn0Au3yMo+z3HuGe17Zi0wO0yiLGGF5cu4+hKTFk9+tldRy/o0Uf4C4fk8YvLhjGstxy/rFyt9VxVIDaUlrDjgNHuX5KP70S1g1cMgWC8m3fnzaQ4qp6Hv+kkP6JPbhyXLrVkVSAeWltCT3CgrlirF4J6w66R68QER6aM5IpAxO4741tbC2tsTqSCiDV9c0syz3AnLFpRIfrvqc7aNErAMJCgnjy+nEkxYRzx5JNHG1ssTqSChCvbyyjqdXG9ZP1IKy7aNGr/4qLCuPxeWM4UNPIL9/cprcuVG7XZjO8tG4f2f16MTxV7wnrLlr06n+M7xfPz2cPYXluOS+vL7E6jvJzK/Mq2He4gZvPGmB1FL+mRa++5QfTBnF2ZiK/f28nBRU646Vyn4WfF5MWF8nsrBSro/g1LXr1LUFBwqPXjKFHeAj3vp6rE6Apt8gtq2H93iPcNLU/IcFaRe6kW1d1KCkmnAcvG8GW0hoWflFkdRzlhxZ+UUx0eIjOa+MBWvSqU5eOSmV2Vgp//3g3RVV1VsdRfqS89jjLc8u5dkIGMRF6M3t306JXnRIR/jBnJBGhwfyfDuEoF1r01T5sxnDjmf2tjhIQtOjVSSX3jOA3l2SRs69az8JRLnG0sYXF6/Zx4chUMuL1xvWeoEWvTunKcWmcOSiBv36Yz+G6JqvjKB/3wld7OdbYyg/PHWR1lIChRa9OSUT4/eUjON7Sxp8/yLc6jvJh9U2tLPyimBnDkhmZFmt1nIChRa+6ZHByDLecNZDXNpaRs/eI1XGUj1q8bh/VDS3cMWOw1VECiha96rKfzhxMn9gIHnh7O61tNqvjKB/T2NLGgs+KOWtwIuP66pzznqRFr7osKiyEX1+SRf7BY3pgVp22pRtKOVTXpHvzFuh20YtIhoisFpE8EdkhInc6Ho8XkRUiUuB4q7+6/cgFI3szeWA8/1hZoDNcqi5rbGnjqTV7mNg/nskDE6yOE3Cc2aNvBX5ujBkOTAZ+LCJZwH3AKmNMJrDK8bHyEyLCAxdnUd3QzL9WF1odR/mI57/aS3ltI3fPGmJ1lIDU7aI3xpQbYzY53j8G5AFpwOXAIsdii4A5zoZU3mVkWixXjk3nuS/2Unqkweo4ystV19t3CmYMS2bKIN2bt4JLxuhFpD8wFlgHpBhjysH+ywBI7uRzbheRHBHJqaqqckUM5UH3nj+UoCB45EM93VKd3L9WF1Lf1MovLhhmdZSA5XTRi0g08AZwlzHmaFc/zxizwBiTbYzJTkpKcjaG8rDesRHcPm0Qy3LL2VRSbXUc5aVKjzTwwtf7+M64dIb2jrE6TsByquhFJBR7yS82xrzpeLhCRFIdz6cClc5FVN7q+9MGkhQTzh+W7dS7UakOPbpiNyLws9k6Nm8lZ866EWAhkGeMebTdU+8C8x3vzwfe6X485c16hIdwz+whbCqpYfm2cqvjKC+zYe8R3tq8n1vOGkBqbKTVcQKaM3v0U4EbgBkissXx7yLgz8AsESkAZjk+Vn7qqvEZDOsdwyMf5tPU2mZ1HOUlmltt/OrNbaTFRep5814gpLufaIz5ApBOnp7Z3a+rfEtwkHD/xcO5YeF6Xlpbwi16708FLPhsDwWVdTx34wSiwrpdM8pF9MpY5bSzM5M4OzORJz4poPa4XkQV6IoP1fP4J4VcfEYq04d1eNKd8jAteuUSv7hgGNUNLTy1Zo/VUZSFjDE88PY2woOD+O2lWVbHUQ5a9MolRqbFcsXYNJ79opjy2uNWx1EWef6rvXxZeJj7LhpGcs8Iq+MoBy165TI/mzUEY+CxFbutjqIskH/wKH/6IJ+Zw5K5bmJfq+OodrTolctkxEfxvSn9eH1jGbsOHrM6jvKgo40t/PClTcRGhvKXq0ZhP/taeQs9HK5c6o4Zg3k1p5RHPszn2RsnWB3Hpxyua2Jn+VFKjxzncF0TLW02wkODiYsKZVBSNENSYojvEWZ1zG+x2Qw/W7qF0iMNvHzbZBKiw62OpE6gRa9cKi4qjB9NH8yfP8jn6z2HdRKrU8g/eJS3Nu1nVX4lhZV1p1w+OSacszITmTEsmbMzk4iNDPVAys4ZY/j9sp2szKvkd5eNYOKAeEvzqI5p0SuXu/HM/iz6ai9//iCPt388Vf+MP4Exhk93VfHvTwvZsLea0GBh0oAErhqfzqi0WPon9iAxOpzQYKG5zcahumYKKo5RUFFH7v5aVuVV8uam/YQECTOHJ3NNdgbnDEkiJNjzI7FPrtnD81/t5eapA/jelH4eX7/qGi165XIRocH8fPZQ7nltK8u3lXPJqD5WR/IaOw7U8vv3drKu+AhpcZE8cPFwrhyX3umQTHhIMGlxkaTFRXLuUPs56a1tNraU1vDh9oO8vWU/H+2oIDkmnLkTMpg3qa/Hphv496eF/OXDXVw2ug8PXDxcf6F7MfGGyaiys7NNTk6O1TGUC7XZDBc//jnHW9pYcfc5hIUE9nH/5lYbj63czYLPioiLDOWu8zKZO7EvoU7uhbe02VidX8krG0pZvauSIBFmDU/he1P6MWVQglvKt81meHh5Hs9+WczlY/rw96tHW/LXhAIR2WiMyT7lclr0yl1W76rkpuc28OClWdw4NXCnRiirbuDHL29ma2kN12Snc/9FWcRGuX5sveRwA4vX7+PVDaVUN7QwODmaGyb348pxacREuGZ9Vcea+PlrW/lsdxU3Te3PAxdnERyke/JW0aJXljPG8N1n1pF/8Bhr7j3XZWXjSz7dVcldS7fQ2mb429WjuGBkqtvX2djSxrLccl5cu4+tpTVEhQVzxdg0bpjSj2G9e3braxpjeGvzfv74fh7HGlv57aUjuG6SnitvNS165RW2ldVy6RNf8JMZg/n57KFWx/Go574s5vfLdjI0JYYnrx/PgMQeHs+QW1bDC1/v472tB2hqtTGxfzzzJmUwLTOpS6dBNrfa+HjnQf6zpoht+2sZnR7LX68ezZAUvYmIN9CiV17jJ0s2s2LnQdbcO52UALgs3hjDoyt28/8+KWR2Vgr/nDuWyLBgSzNV1zfz2sZSXlpbQonjPr9ZqT2ZPDCBYb1j6BMXSc/IEGzGvuzew/VsKqlhza5Kjja2MiCxBz86dxDfGZdOkA7VeA0teuU1Sg43MPPRT7lqfDp/unKU1XHcqs1meODt7SxZX8LcCRn8Yc5IrzpQabMZcvfX8kVBFV8UHmJTSQ3NrbYOl03pGc7UwYlcOroP0zLF0uQpAAAMZ0lEQVSTdCzeC3W16PX0SuV2fROiuH5yPxZ9tZdbzhrA4GT//LPfZjP83+u5vLGpjB9PH8Q9s4d63SmHQUHCmIw4xmTEcceMTNpshtIjDVQea+Lo8RaCgiA2Moz0XpEkx4R7XX7VPVr0yiN+MiOT13PK+PMHu3hm/il3QHyOMYb7397OG5vKuPu8Idx5XqbVkbokOEjon9iD/hYcP1Ce4z1/Uyq/Ft8jjB+cO4iVeRWsLz5idRyXMsbwu/d2smR9CT+ePoifztRb5ynvokWvPObmqQPo3TOCP32QhzccG3KVx1bs5vmv9nLrWQO8crhGKS165TGRYcHcPSuTzSX2y/f9wcvrSnj8k0L7hVA6DYDyUlr0yqO+My6dISnR/OWjXbS0dXy2h69YubOCB97exvShSTx8xRla8spradErjwoJDuK+C4dRfKie574stjpOt20uqeaOJZsYmRbLE9eNc3rOGqXcSV+dyuNmDEvhvOHJ/GNlgU/eX7b4UD23LMohpWcEz944gR7hevKa8m5a9MoSv710BDZjeGjZTqujnJZDdU3Mf3Y9Aiy6aSKJejcl5QO06JUlMuKjuGP6YN7fdpA1u6usjtMlx5vbuHVRDpXHGnlmfraee658hluKXkQuEJFdIlIoIve5Yx3K9902bSADE3vw23e209jSZnWck2qzGe5aupmtZTX8c+5YxvbtZXUkpbrM5UUvIsHAv4ALgSxgnohkuXo9yveFhwTz0JyR7D3cwGMrd1sd56T++H4eH+2o4NcXZ3H+iN5Wx1HqtLhjj34iUGiMKTLGNAOvAJe7YT3KD0wdnMi8iRk8/VkRG/dVWx2nQ89/WczCL4q5aWp/bj4rcG+gonyXO4o+DSht93GZ4zGlOnT/xVmkxkZyz2tbOd7sXUM4H+84yO+W7eT8ESk8cLH+Yap8kzuKvqOrRr51vbuI3C4iOSKSU1XlGwfjlHtEh4fw16tHUXyonr9+tMvqOP+1pbSGn76ymVHpcfzj2rE6Ta/yWe4o+jIgo93H6cCBExcyxiwwxmQbY7KTkpLcEEP5kjMHJTJ/Sj+e+6qYLwoOWR2H4kP13LpoA0kx4Sycn235jUOUcoY7in4DkCkiA0QkDJgLvOuG9Sg/84sLh5GZHM1PX9nMgRrrLqQ6UHOc659ZhzHwvJ4rr/yAy4veGNMK3AF8BOQBrxpjdrh6Pcr/RIWF8OT142lutfHDxZtoavX8eP3huiauX7iOo8dbWHTzRAYlRXs8g1Ku5pbz6I0x7xtjhhhjBhljHnbHOpR/GpQUzd+uHsXW0hp+/55nr5qtaWhm/nPr2V99nIU3TmBkWqxH16+Uu+iVscrrXDAyle+fM5DF60p4ae0+j6zzUF0TcxesZffBOp66YTwTB8R7ZL1KeYLOxqS80r2zh1JQUcdv3tlOUky4Wy9SOljbyHefWcv+muMsvDGbszP15ADlX3SPXnmlkOAgnrhuLKPS4/jJy5v5JL/CLesprKzjmv98TcXRJl64eZKWvPJLWvTKa0WFhbDopokM7R3DD17cxIfby1369VfsrGDOv76kobmVl26dpMM1ym9p0SuvFhsVyku3TmJkWk9+uHgT/1mzx+n7zbbZDP9YuZvbXshhYFIP3r3jLMZkxLkosVLeR4teeb3YyFBevm0yF41M5U8f5HPbCxs5XNfUra/1zVDNP1YW8J1x6bz6/Sn0iYt0cWKlvIsejFU+ISI0mCeuG8v4L3vxpw/ymPH3NdwzewjXTMggPOTUV60erG3kqTV7eGntPnqEh/DoNaO5Ymya3udVBQRx9s9gV8jOzjY5OTlWx1A+oqDiGL95ZwdfFx0mKSacq8anc/6I3gxPjfmf0q861sT64iO8v62cj3YcxGYMcyf25e7zhpAUo1e7Kt8nIhuNMdmnXE6LXvkiYwxf7TnM058X8XnBIdpshtBgITE6nLCQIGqPt1DT0AJAQo8wLhvTh5vOHEDfhCiLkyvlOl0teh26UT5JRJg6OJGpgxOprm/myz2H2HngKFXHmmhqtREbGUq/hCjGZMQxtm8vnXlSBTQteuXzevUI45JRfbhkVB+royjllfSsG6WU8nNa9Eop5ee06JVSys9p0SullJ/ToldKKT+nRa+UUn5Oi14ppfycFr1SSvk5r5gCQUSqgO7eMy4ROOTCOK7krdk01+nRXKfPW7P5W65+xphT3i3HK4reGSKS05W5Hqzgrdk01+nRXKfPW7MFai4dulFKKT+nRa+UUn7OH4p+gdUBTsJbs2mu06O5Tp+3ZgvIXD4/Rq+UUurk/GGPXiml1En4RNGLyNUiskNEbCKSfcJzvxSRQhHZJSLnd/L5A0RknYgUiMhSEQlzQ8alIrLF8W+viGzpZLm9IrLNsZxHbqslIg+KyP52+S7qZLkLHNuxUETu80Cuv4pIvojkishbIhLXyXIe2Wan+v5FJNzxcy50vJ76uytLu3VmiMhqEclz/B+4s4NlzhWR2nY/39+4O1e7dZ/0ZyN2jzu2Wa6IjPNApqHttsUWETkqInedsIxHtpmIPCsilSKyvd1j8SKywtFHK0SkVyefO9+xTIGIzHcqiDHG6/8Bw4GhwKdAdrvHs4CtQDgwANgDBHfw+a8Ccx3vPwX80M15/w78ppPn9gKJHt5+DwL3nGKZYMf2GwiEObZrlptzzQZCHO8/Ajxi1TbryvcP/Ah4yvH+XGCpB352qcA4x/sxwO4Ocp0LLPPka6qrPxvgIuADQIDJwDoP5wsGDmI/39zj2wyYBowDtrd77C/AfY737+vodQ/EA0WOt70c7/fqbg6f2KM3xuQZY3Z18NTlwCvGmCZjTDFQCExsv4CICDADeN3x0CJgjruyOtZ3DbDEXetwk4lAoTGmyBjTDLyCffu6jTHmY2NMq+PDtUC6O9d3Cl35/i/H/voB++tppuPn7TbGmHJjzCbH+8eAPCDNnet0scuBF4zdWiBORFI9uP6ZwB5jTHcvyHSKMeYz4MgJD7d/HXXWR+cDK4wxR4wx1cAK4ILu5vCJoj+JNKC03cdlfPs/QQJQ065QOlrGlc4GKowxBZ08b4CPRWSjiNzuxhwnusPxp/Oznfyp2JVt6U43Y9/z64gntllXvv//LuN4PdVif315hGOoaCywroOnp4jIVhH5QERGeCoTp/7ZWP26mkvnO11WbbMUY0w52H+RA8kdLOPS7eY194wVkZVA7w6eut8Y805nn9bBYyeeRtSVZbqkixnncfK9+anGmAMikgysEJF8x299p5wsG/Ak8BD27/sh7ENLN5/4JTr4XKdPyerKNhOR+4FWYHEnX8Yt2+zEqB085rbX0ukSkWjgDeAuY8zRE57ehH1oos5x/OVtINMTuTj1z8bKbRYGXAb8soOnrdxmXeHS7eY1RW+MOa8bn1YGZLT7OB04cMIyh7D/uRji2AvraBmXZBSREOBKYPxJvsYBx9tKEXkL+5CB06XV1e0nIk8Dyzp4qivb0uW5HAeZLgFmGsfgZAdfwy3b7ARd+f6/WabM8bOO5dt/lruciIRiL/nFxpg3T3y+ffEbY94XkX+LSKIxxu1zunThZ+OW11UXXQhsMsZUnPiEldsMqBCRVGNMuWMYq7KDZcqwH0f4Rjr2Y5Td4utDN+8Ccx1nQwzA/ht5ffsFHOWxGrjK8dB8oLO/EJx1HpBvjCnr6EkR6SEiMd+8j/1g5PaOlnWlE8ZEr+hknRuATLGfoRSG/U/ed92c6wLgF8BlxpiGTpbx1Dbryvf/LvbXD9hfT5909svJVRzHABYCecaYRztZpvc3xwpEZCL2/9eH3ZnLsa6u/GzeBb7nOPtmMlD7zbCFB3T617VV28yh/euosz76CJgtIr0cQ62zHY91j7uPOrviH/ZyKgOagArgo3bP3Y/9bIldwIXtHn8f6ON4fyD2XwCFwGtAuJtyPg/84ITH+gDvt8ux1fFvB/bhC09svxeBbUCu40WWemI2x8cXYT+rY48nsjl+HqXAFse/p07M5clt1tH3D/we+y8igAjH66fQ8Xoa6IFtdBb2P9lz222ni4AffPNaA+5wbJut2A9qn+mh11WHP5sTsgnwL8c23Ua7s+bcnC0Ke3HHtnvM49sM+y+acqDF0WG3YD+uswoocLyNdyybDTzT7nNvdrzWCoGbnMmhV8YqpZSf8/WhG6WUUqegRa+UUn5Oi14ppfycFr1SSvk5LXqllPJzWvRKKeXntOiVUsrPadErpZSf+/8T35abToeyigAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def f(x):\n",
    "    return x**2 + 10*np.sin(x)\n",
    "\n",
    "x = np.arange(-10, 10, 0.1)\n",
    "plt.plot(x, f(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now find the minimum with a few methods\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      fun: -7.945823375615215\n",
      " hess_inv: array([[0.08589237]])\n",
      "      jac: array([-1.1920929e-06])\n",
      "  message: 'Optimization terminated successfully.'\n",
      "     nfev: 18\n",
      "      nit: 5\n",
      "     njev: 6\n",
      "   status: 0\n",
      "  success: True\n",
      "        x: array([-1.30644012])\n"
     ]
    }
   ],
   "source": [
    "from scipy import optimize\n",
    "\n",
    "# The default (Nelder Mead)\n",
    "print(optimize.minimize(f, x0=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      fun: array([-7.94582338])\n",
      " hess_inv: <1x1 LbfgsInvHessProduct with dtype=float64>\n",
      "      jac: array([-1.42108547e-06])\n",
      "  message: b'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'\n",
      "     nfev: 12\n",
      "      nit: 5\n",
      "   status: 0\n",
      "  success: True\n",
      "        x: array([-1.30644013])\n"
     ]
    }
   ],
   "source": [
    "print(optimize.minimize(f, x0=0, method=\"L-BFGS-B\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![img](img/optm1.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.optimize import minimize\n",
    "\n",
    "def objective(x):\n",
    "    return x[0]*x[3]*(x[0]+x[1]+x[2])+x[2]\n",
    "\n",
    "def constraint1(x):\n",
    "    return x[0]*x[1]*x[2]*x[3]-25.0\n",
    "\n",
    "def constraint2(x):\n",
    "    sum_eq = 40.0\n",
    "    for i in range(4):\n",
    "        sum_eq = sum_eq - x[i]**2\n",
    "    return sum_eq\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initial guesses\n",
    "x0 = np.array([1.0,5.0,5.0,1.0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial Objective: 16.0\n"
     ]
    }
   ],
   "source": [
    "# show initial objective\n",
    "print('Initial Objective: ' + str(objective(x0)))\n",
    "\n",
    "# optimize\n",
    "b = (1.0,5.0)\n",
    "bnds = (b, b, b, b)\n",
    "con1 = {'type': 'ineq', 'fun': constraint1}\n",
    "con2 = {'type': 'eq', 'fun': constraint2}\n",
    "cons = ([con1,con2])\n",
    "\n",
    "solution = minimize(objective,x0,method='SLSQP',\\\n",
    "                    bounds=bnds,constraints=cons)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final Objective: 17.01401724563517\n",
      "Solution\n",
      "x1 = 1.0\n",
      "x2 = 4.742996096883977\n",
      "x3 = 3.8211546234095715\n",
      "x4 = 1.379407645075325\n"
     ]
    }
   ],
   "source": [
    "x = solution.x\n",
    "\n",
    "# show final objective\n",
    "print('Final Objective: ' + str(objective(x)))\n",
    "\n",
    "# print solution\n",
    "print('Solution')\n",
    "print('x1 = ' + str(x[0]))\n",
    "print('x2 = ' + str(x[1]))\n",
    "print('x3 = ' + str(x[2]))\n",
    "print('x4 = ' + str(x[3]))\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}