Opt landing updates

local_optimization/BXNL/Readme.md

@@ -1,10 +1,13 @@
-[![NAG](https://raw.githubusercontent.com/numericalalgorithmsgroup/NAGPythonExamples/master/nag_logo.png)](https://www.nag.co.uk)
+[![NAG Logo](../../nag_logo.png)](https://www.nag.com)
 
 # Nonlinear Least-Squares Trust-Region Method (BXNL)
 
+[[`handle_solve_bxnl`](https://www.nag.co.uk/numeric/py/nagdoc_latest/naginterfaces.library.opt.html#naginterfaces.library.opt.handle_solve_bxnl) | [`e04ggf`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04ggf.html) | 
+[`e04ggc`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/clhtml/e04/e04ggc.html) ]
+
 Data fitting and calibrating parameters of complex numerical models is one of the most common
 problems found in numerous industries such as  physics, space exploration, simulations, engineering, amongs many others. 
-[NAG](https://www.nag.co.uk/) introduces to the [NAG Library at Mark 27.1](https://www.nag.co.uk/content/nag-library) a novel [nonlinear least-square](https://en.wikipedia.org/wiki/Non-linear_least_squares) [trust-region solver](https://en.wikipedia.org/wiki/Trust_region) for unconstrained or bound-constrained fitting problems, [`handle_solve_bxnl`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/clhtml/e04/e04ggc.html) ([`e04gg`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04ggf.html)). It offers a significant variety of algorithms and regularisation techniques.
+[NAG](https://www.nag.co.uk/) introduces to the [NAG Library at Mark 27.1](https://www.nag.co.uk/content/nag-library) a novel [nonlinear least-square](https://en.wikipedia.org/wiki/Non-linear_least_squares) [trust-region solver](https://en.wikipedia.org/wiki/Trust_region) for unconstrained or bound-constrained fitting problems, [`handle_solve_bxnl`](https://www.nag.co.uk/numeric/py/nagdoc_latest/naginterfaces.library.opt.html#naginterfaces.library.opt.handle_solve_bxnl) ([`e04gg`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04ggf.html)). It offers a significant variety of algorithms and regularisation techniques.
 
 The solver [`e04gg`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04ggf.html) is aimed at small to medium sized fitting problems (up to 1000s of parameters) bound-constrained nonlinear least-squares problems 
 and is also part of the [NAG Optimization Modelling Suite](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04intro.html#optsuite) common handle interface. It offers clarity and consistency of the interface of the solvers within the suite, making it trivial to switch among compatible solvers.
@@ -25,7 +28,7 @@ Figure 1 shows an illustrative simple problem of data fitting ([Jupyter Notebook
 # More Info
  1. [BXNL information leaflet](https://www.nag.com/content/faster-data-fitting-solver)
  2. [BXNL in the NAG Library for Python](https://www.nag.co.uk/numeric/py/nagdoc_latest/naginterfaces.library.opt.html#naginterfaces.library.opt.handle_solve_bxnl)
- 3. [BXNL documentation page](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04ggf.html) [[Python example](https://www.nag.co.uk/numeric/py/nagdoc_latest/naginterfaces.library.opt.html#naginterfaces.library.examples.opt.handle_disable_ex.main), [C example](https://www.nag.co.uk/numeric/nl/nagdoc_latest/clhtml/e04/e04ggc.html#example), [Fortran example](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04ggf.html#example)]
+ 3. Examples [[Python example](https://www.nag.co.uk/numeric/py/nagdoc_latest/naginterfaces.library.opt.html#naginterfaces.library.examples.opt.handle_disable_ex.main), [C example](https://www.nag.co.uk/numeric/nl/nagdoc_latest/clhtml/e04/e04ggc.html#example), [Fortran example](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04ggf.html#example)]
 
  # Unfolding Nuclear Track Data
 
@@ -96,4 +99,10 @@ represent less functions and gradients calls.
  * Adachi S, Iwata S, Nakatsukasa Y, and Takeda A (2015) _Solving the trust region subproblem by a generalized eigenvalue problem_. Technical report, METR 2015-14. Mathematical Engineering, The University of Tokyo https://www.keisu.t.u-tokyo.ac.jp/data/2015/METR15-14.pdf
  * Conn A R, Gould N I M and Toint Ph L (2000) _Trust Region Methods_. SIAM, Philadephia
 
+
 <!-- foot banner for commercial material -->
+
+# Obtaining the NAG Library for Python
+
+ * Instructions on [how to install the NAG Library for Python](../Readme.md#install)
+ * Instructions on [how to run the Jupyter notebooks in the Repository](../Readme.md#jupyter)

local_optimization/DFO/Readme.md

@@ -0,0 +1,89 @@
+[![NAG Logo](../../nag_logo.png)](https://www.nag.com)
+
+# Derivative-Free Optimization ([DFO](https://www.nag.com/numeric/nl/nagdoc_latest/flhtml/e04/e04intro.html#derivatives))
+
+[DFO](https://www.nag.com/numeric/nl/nagdoc_latest/flhtml/e04/e04intro.html#derivatives) solvers are aimed at optimizing _black box_ models and can handle either [calibration (nonlinear least squares)](https://en.wikipedia.org/wiki/Non-linear_least_squares) problems (DFLS) 
+or [problems with a generic objective function](https://en.wikipedia.org/wiki/Nonlinear_programming) (DFNO).
+
+* Calibration: DFLS (Derivative Nonlinear least squares)
+[[`handle_solve_dfls`](https://www.nag.co.uk/numeric/py/nagdoc_latest/naginterfaces.library.opt.html#naginterfaces.library.opt.handle_solve_dfls) | 
+[`e04fff`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04fff.html) | 
+[`e04ffc`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/clhtml/e04/e04ffc.html) ]
+
+ * DFNO (Derivative-Free Nonlinear Optimization) 
+ [[`handle_solve_dfno`](https://www.nag.co.uk/numeric/py/nagdoc_latest/naginterfaces.library.opt.html#naginterfaces.library.opt.handle_solve_dfno) | 
+ [`e04jdf`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04jdf.html) | 
+[`e04jdc`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/clhtml/e04/e04jdc.html) ]
+
+
+Optimizing complex numerical models is one of the most common problems found in the industry (finance, multi-physics simulations, 
+engineering, etc.). To solve these optimization problems with a standard optimization algorithm such as 
+[Gauss–Newton](https://en.wikipedia.org/wiki/Gauss%E2%80%93Newton_algorithm) (for 
+problems with a nonlinear least squares structure) or 
+[CG](https://en.wikipedia.org/wiki/Conjugate_gradient_method) (for unstructured nonlinear objective) requires good estimates 
+of the model's derivatives. 
+If exact derivatives are easy to compute then using derivative-based methods is preferable. However, explicitly writing the derivatives 
+or applying [AD methods](https://www.nag.com/content/algorithmic-differentiation-software) might be impossible if the model is a black box. 
+The alternative, estimating derivatives via [finite differences](https://en.wikipedia.org/wiki/Finite_difference#Relation_with_derivatives), 
+can quickly become impractical or too computationally expensive. Under these circumstances, an attractive optimization solver that does not 
+require the user to provide any derivatives is the model-based DFO solver.
+
+NAG's model-based [DFO](https://www.nag.com/numeric/nl/nagdoc_latest/flhtml/e04/e04intro.html#derivatives) [solvers for DFLS and DFNO]() present a number of attractive features:
+
+ * Proved resilient to noise,
+ * The least-square solver is able to start making progress with as few as two objective evaluations,
+ * Integrated to the [NAG Optimization Modeling Suite (NOMS)](https://www.nag.com/numeric/nl/nagdoc_latest/clhtml/e04/e04intro.html#optsuite) with simple interfaces for the solvers and related routines,
+ * Optional reverse communication interface.
+
+![2 steps of DFO algorithm](animation.gif)
+
+**Figure 1.** Animation showing 2 iterations of a model-based DFO algorithm [`handle_solve_dfls`](https://www.nag.co.uk/numeric/py/nagdoc_latest/naginterfaces.library.opt.html#naginterfaces.library.opt.handle_solve_dfls).
+
+
+## Poster
+<table><tr>
+<td valign="top">A 2019 poster discussing NAG's DFO/DFLS functionality 
+<a href="https://www.nag.com/market/posters/derivative_free_optimization_solver_calibration_problems.pdf">is available on the NAG website</a>.</td>
+<!--- td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</td --->
+<td><a href="https://www.nag.com/market/posters/derivative_free_optimization_solver_calibration_problems.pdf">
+<img src="https://www.nag.com/sites/default/files/styles/paragraph_image_/public/2020-01/dfo-solver_1.png?itok=6_PijS2l" 
+width="500px" alt="DFO Poster thumbnail"/></a></td>
+</tr></table>
+
+## Example 
+
+The Jupyter notebook showcases the optimization of noisy problems where the objective function is not deterministic. 
+The example discuses and illustrates the advantages of using a DFO solver instead of a derivative-based solver using 
+finite difference estimations for the gradient.
+
+  * [Noisy problem notebook.](DFO_noisy.ipynb)
+
+## More information 
+
+ * [Informative Leaflet](https://www.nag.com/content/derivative-free-optimization-dfo)
+
+ * Blog post from the OptCorner [The price of derivatives - Derivative-free Optimization](https://www.nag.com/blog/optcorner-price-derivatives-derivative-free-optimization)
+
+ * [DFO/DFLS in the NAG Library for Python](https://www.nag.co.uk/numeric/py/nagdoc_latest/naginterfaces.library.opt.html#naginterfaces.library.opt.handle_solve_dfls)
+
+ * Examples [ [Python example](https://www.nag.com/numeric/py/nagdoc_latest/naginterfaces.library.opt.html#naginterfaces.library.examples.opt.handle_solve_dfls_ex.main) | [C example](https://www.nag.co.uk/numeric/nl/nagdoc_latest/clhtml/e04/e04ffc.html#example) | [Fortran 90 example](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04fff.html#example) ]
+
+ * [DFO/DFNO in the NAG Library for Python](https://www.nag.co.uk/numeric/py/nagdoc_latest/naginterfaces.library.opt.html#naginterfaces.library.opt.handle_solve_dfno)
+
+ * Examples [ [Python example](https://www.nag.com/numeric/py/nagdoc_latest/naginterfaces.library.opt.html#naginterfaces.library.examples.opt.handle_solve_dfno_ex.main) | [C example](https://www.nag.co.uk/numeric/nl/nagdoc_latest/clhtml/e04/e04jdc.html#example) | [Fortran 90 example](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04jdf.html#example) ]
+
+
+## References
+
+* C. Cartis, J. Fiala, B. Marteau, and L. Roberts (2019) _Improving the Flexibility and robustness of 
+  model-based derivative-free optimization solvers_. ACM Transactions On Numerical Software.
+* C. Cartis and L. Roberts (2017) _A derivative-free Gauss–Newton method_. Mathematical Programming Computation.
+* Powell M. J. D. (2009) _The BOBYQA algorithm for bound constrained optimization without derivatives_. Report DAMTP 2009/NA06 University of Cambridge.
+
+<!-- foot banner for commercial material -->
+
+# Obtaining the NAG Library for Python
+
+ * Instructions on [how to install the NAG Library for Python](../Readme.md#install)
+ * Instructions on [how to run the Jupyter notebooks in the Repository](../Readme.md#jupyter)
+

local_optimization/FOAS/README.md

@@ -1,6 +1,8 @@
-[![NAG](https://raw.githubusercontent.com/numericalalgorithmsgroup/NAGPythonExamples/master/nag_logo.png)](https://www.nag.co.uk)
+[![NAG Logo](../../nag_logo.png)](https://www.nag.com)
 
-# First-order active-set method (FOAS) [`handle_solve_bounds_foas (e04kf)`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04kff.html)
+# First-order active-set method (FOAS)
+[[`handle_solve_bounds_foas`](https://www.nag.co.uk/numeric/py/nagdoc_latest/naginterfaces.library.opt.html#naginterfaces.library.opt.handle_solve_bounds_foas) | [`e04kff`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04kff.html) | 
+[`e04kfc`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/clhtml/e04/e04kfc.html) ]
 
 Implementations of first-order methods not only are ubiquitous and have a widespread use, they have also demonstrated to endure the challenges of ever-growing problems sizes imposed by the industry. Most notable are applications in statistics, e.g. parameter calibration for log-linear models, conditional random fields (L2-regularisation) or logistic multi-class regression, amongs many other. First-order methods and the Conjugate Gradient method inparticular have been a research subject for well over 50 years and continue to be improved.
 
@@ -17,13 +19,13 @@ The following example illustrates the simple usage of FOAS to solve the bound-co
 
 **Figure 1.** 2D Rosenbrock function, (left) the minimum is shown as a yellow dot at x=(1,1). On the right a bound constrained version showing with a purple dotted line a path towards the constrained solution point on the border.
 
-# More information 
+## More information 
  1. [FOAS information page](https://www.nag.com/content/limited-memory-nonlinear-conjugate-gradient-solver)
  2. [FOAS in the NAG Library for Python](https://www.nag.co.uk/numeric/py/nagdoc_latest/naginterfaces.library.opt.html#naginterfaces.library.opt.handle_solve_bounds_foas)
  3. [FOAS documentation page](https://www.nag.co.uk/numeric/nl/nagdoc_latest/clhtml/e04/e04kfc.html) [ [C](https://www.nag.co.uk/numeric/nl/nagdoc_latest/clhtml/e04/e04kfc.html) | [Fortran](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04kff.html) ]
  4. Examples [ [Python example](https://www.nag.co.uk/numeric/py/nagdoc_latest/naginterfaces.library.opt.html#naginterfaces.library.examples.opt.handle_solve_bounds_foas_ex.main) | [C example](https://www.nag.co.uk/numeric/nl/nagdoc_latest/clhtml/e04/e04kfc.html#example) | [Fortran example](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04kff.html#example) ]
 
-# A modern replacement for NAG solver [`uncon_conjgrd_comp` (`e04dg`)](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04dgf.html)
+## A modern replacement for NAG solver [`uncon_conjgrd_comp` (`e04dg`)](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04dgf.html)
 One of the main design objectives for `handle_solve_bounds_foas` (`e04kf`) was to provide a modern and attractive replacement for the CG solver `e04dg` introduced in Mark 12. While this solver was targeted for unconstrained NLPs, `e04kf` has been extended with an active-set method in order to solve bound-constrained NLPs.
 
 More recent and modern methods have been incorporated into `e04kf` making it much faster than `e04dg`. The following Figure 2 reports performance profiles over 114 unconstrained NLP CUTEst problems for both solvers `e04kf` and `e04dg`. Contrasting the three plots, it is evident that the new solver is more efficient in time (40% faster) and in general terms is less expensive: requires less function and gradient evaluations.
@@ -36,15 +38,16 @@ More recent and modern methods have been incorporated into `e04kf` making it muc
 
 **Figure 2.** Performance profiles comparing solvers `e04kf` and `e04dg`. In the time plot on the left, higher line indicates faster solver. For the center and right plots higher line represent less functions (NF) or gradients (NG) calls. For all three plots it can be seen that `e04kf` is 40% faster in time and requires less function and gradient calls.
 
-# Migrating from [`uncon_conjgrd_comp (e04dg)`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04dgf.html) (Mark 26.x) to [`handle_solve_bounds_foas (e04kf)`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04kff.html) (Mark 27+)
+## Migrating from Marks 25 and 26 to Mark 27
 
-Notes and comments on migrating your code to the new FOAS solver:
+Notes and comments on migrating your code from [`uncon_conjgrd_comp (e04dg)`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04dgf.html) to the new FOAS solver [`handle_solve_bounds_foas`](https://www.nag.co.uk/numeric/py/nagdoc_latest/naginterfaces.library.opt.html#naginterfaces.library.opt.handle_solve_bounds_foas) ([`e04kff`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/flhtml/e04/e04kff.html), 
+[`e04kfc`](https://www.nag.co.uk/numeric/nl/nagdoc_latest/clhtml/e04/e04kfc.html)):
 
  * [Python](migration/migration_e04dg_e04kf.ipynb)
  * [Fortran 90](https://www.nag.com/numeric/nl/nagdoc_latest/flhtml/genint/replace.html#e04dgf)
  * [C](https://www.nag.com/numeric/nl/nagdoc_latest/clhtml/genint/replace.html#e04dgc)
 
-# Beale's function
+## Beale's function
 This example compares the steps taken by FOAS and L-BFGS-B 3.0 to find the solution point to [Beale's function](https://en.wikipedia.org/wiki/Test_functions_for_optimization). It is a classic nonconvex test function used to benchmark nonlinear optimization solvers.
 
 Both solvers are used to find a minimum to the function and are started at the same initial point (2, 2). The following figure shows an animation of the steps taken by each solver to find a minimum to the function. 
@@ -54,7 +57,7 @@ It illustrates the agressive steps taken by the [Conjugate Gradient method](http
 
 **Figure 3.** Contour plots for Beale's function (thin blue lines), and the steps taken by FOAS (red) and L-BFGS-B 3.0 (blue) to find the minimum of Beale's funtion at (3, 0.5) marked with a magenta star. It can be seen that FOAS by the 8th step provides a reasonable approximation to the solution point while L-BFGS-B is still relatively far from it. 
 
-# References
+## References
 
  * Hager W W and Zhang H (2005) _A New Conjugate Gradient Method with Guaranteed Descent and an Efficient Line Search_. SIAM J. Optim. 16(1) 170–192
  * Hager W W and Zhang H (2006a) _Algorithm 851: CG DESCENT, a Conjugate Gradient Method with Guaranteed Descent_. ACM Trans. Math. Software 32(1) 113–137
@@ -64,4 +67,8 @@ It illustrates the agressive steps taken by the [Conjugate Gradient method](http
 
 <!-- foot banner for commercial material -->
 
+# Obtaining the NAG Library for Python
+
+ * Instructions on [how to install the NAG Library for Python](../Readme.md#install)
+ * Instructions on [how to run the Jupyter notebooks in the Repository](../Readme.md#jupyter)
 

local_optimization/Modelling/Readme.md

@@ -0,0 +1,52 @@
+[![NAG Logo](../../nag_logo.png)](https://www.nag.com)
+
+# Modelling Optimization Problems
+
+In this folder you will find notebook examples, tips and tricks related to modeling optimization problems.
+
+* **Christmas Special** [How to decorate your Christmas tree (Notebook)](christmas_demo.ipynb) 
+   or view the [Blog-post](https://www.nag.com/blog/optcorner-christmas-edition).
+<table><tr>
+
+<td><a href="christmas_demo.ipynb">
+<img src="../images/xmas_tree.png" 
+width="100" height="100px" alt="Xmas Tree Plot"/></a></td>
+
+<td valign="top">See how optimization can help to decorate a Christmas tree...</td>
+</tr></table>  
+
+
+* **Demo** [Linear Programming (LP)](LP_demo.ipynb).
+
+   Tiny and Cute LP problem...
+* **Demo** [Nonlinear calibration (data fitting)](handle_disable_ex.ipynb). 
+<table><tr>
+<td><a href="handle_disable_ex.ipynb">
+<img src="../images/nlls.png" 
+width="100" height="80px" alt="Data fitting Plot"/></a></td>
+
+<td valign="top">This demo shows useful features of the <a href="https://www.nag.com/numeric/nl/nagdoc_latest/flhtml/e04/e04intro.html#optsuite">NAG Optimization Modelling Suite (NOMS)</a> in a Nonlinear Least-squares problem.</td>
+
+</tr></table>  
+
+
+* **Demo** [Production Planning](production_planning.ipynb).
+
+   Optimal production planning showcasing features of the [NAG Optimization Modelling Suite (NOMS)](https://www.nag.com/numeric/nl/nagdoc_latest/flhtml/e04/e04intro.html#optsuite).
+
+# Useful Links
+* [Background to Optimization Methods](https://www.nag.com/numeric/nl/nagdoc_latest/flhtml/e04/e04intro.html#algorithms)
+* [Nag Blog](https://www.nag.com/content/nag-blog)
+
+<!-- When ready add links?
+* Blog-post: Introducing the NAG Optimization Modelling Suite (NOMS)
+* The right tool for the job I – matching problem with optimizer
+* Blog-post: The right tool for the job II - dense vs. sparse
+--->
+
+<!-- foot banner for commercial material -->
+
+# Obtaining the NAG Library for Python
+
+ * Instructions on [how to install the NAG Library for Python](../Readme.md#install)
+ * Instructions on [how to run the Jupyter notebooks in the Repository](../Readme.md#jupyter)