Merge pull request #542 from fitbenchmarking/release-0.1.x

Merging release branch into master
fitbenchmarking · May 22, 2020 · 8c0eb0d · 8c0eb0d
2 parents 937d2c4 + 120b184
commit 8c0eb0d
Show file tree

Hide file tree

Showing 33 changed files with 692 additions and 343 deletions.
diff --git a/.coveragerc b/.coveragerc
@@ -1,6 +1,7 @@
 [run]
 omit =
     */tests/*
+    fitbenchmarking/cli/main.py
 
 
 [report]

diff --git a/azure-pipelines.yml b/azure-pipelines.yml
@@ -26,6 +26,10 @@ steps:
 
 - script: |
     pip install pytest pytest-azurepipelines
-    pytest fitbenchmarking\utils\tests\test_logger.py
-#    pytest fitbenchmarking\utils\tests\test_fitbm_result.py (breaks in first azure pipeline install)
+    pytest fitbenchmarking\utils
+    pytest fitbenchmarking\core
+    pytest fitbenchmarking\results_processing
+    pytest fitbenchmarking\parsing
+    pytest fitbenchmarking\controllers\test\test_default_controllers.py
+    pytest fitbenchmarking\cli\tests\test_exception_handler.py
   displayName: 'pytest utils tests'
diff --git a/docs/source/concept/how/Minimizers.rst b/docs/source/concept/how/Minimizers.rst
@@ -16,17 +16,17 @@ It came out of the University of Maryland and NIST as part of the DANSE
 FitBenchmarking currently supports the Bumps minimizers:
 
 * `Nelder-Mead Simplex <https://bumps.readthedocs.io/en/latest/guide/optimizer.html#nelder-mead-simplex>`_ (:code:`amoeba`)
-  
+
 * `Levenberg-Marquardt <https://bumps.readthedocs.io/en/latest/guide/optimizer.html#fit-lm>`_  (:code:`lm`)
-  
+
 * `Quasi-Newton BFGS <https://bumps.readthedocs.io/en/latest/guide/optimizer.html#quasi-newton-bfgs>`_ (:code:`newton`)
-  
+
 * `Differential Evolution <https://bumps.readthedocs.io/en/latest/guide/optimizer.html#differential-evolution>`_ (:code:`de`)
-  
+
 * `MINPACK <https://github.com/bumps/bumps/blob/96b5100fc3d5b9485bd4a444c83a33617b74aa9d/bumps/mpfit.py>`_ (:code:`mp`)  This is a translation of `MINPACK` to Python.
 
 **Links** `GitHub - bumps <https://github.com/bumps/bumps>`_
-  
+
 DFO
 ---
 
@@ -38,7 +38,7 @@ of Oxford, in conjunction with NAG.  They are particularly well suited for solvi
 FitBenchmarking currently supports the DFO minimizers:
 
 * `Derivative-Free Optimizer for Least Squares <http://people.maths.ox.ac.uk/robertsl/dfols/userguide.html>`_ (:code:`dfols`)
-  
+
 * `Derivative-Free Gauss-Newton Solver <http://people.maths.ox.ac.uk/robertsl/dfols/userguide.html>`_ (:code:`dfogn`)
 
  **Links** `GitHub - DFO-GN <https://github.com/numericalalgorithmsgroup/dfogn>`_ `GitHub - DFO-LS <https://github.com/numericalalgorithmsgroup/dfols>`_
@@ -51,7 +51,7 @@ provides a wide range of mathematical routines.  We call GSL using  the `pyGSL P
 <https://sourceforge.net/projects/pygsl/>`_.
 
 The GSL routines have a number of parameters that need to be chosen, often without default suggestions.
-We have taken the values as used by Mantid. 
+We have taken the values as used by Mantid.
 
 We provide implementations for the following
 packages in the `multiminimize <https://www.gnu.org/software/gsl/doc/html/multimin.html>`_ and `multifit <https://www.gnu.org/software/gsl/doc/html/nls.html>`_ sections of the library:
@@ -60,19 +60,19 @@ packages in the `multiminimize <https://www.gnu.org/software/gsl/doc/html/multim
 * `Levenberg-Marquardt (unscaled) <http://pygsl.sourceforge.net/api/pygsl.html#pygsl.multifit_nlin.lmder>`_ (:code:`lmder`)
 
 * `Levenberg-Marquardt (scaled) <http://pygsl.sourceforge.net/api/pygsl.html#pygsl.multifit_nlin.lmsder>`_ (:code:`lmsder`)
-  
+
 * `Nelder-Mead Simplex Algorithm <http://pygsl.sourceforge.net/api/pygsl.html#pygsl.multiminimize.nmsimplex>`_ (:code:`nmsimplex`)
 
 * `Nelder-Mead Simplex Algorithm (version 2) <http://pygsl.sourceforge.net/api/pygsl.html#pygsl.multiminimize.nmsimplex2>`_ (:code:`nmsimplex2`)
-  
+
 * `Polak-Ribiere Conjugate Gradient Algorithm <http://pygsl.sourceforge.net/api/pygsl.html#pygsl.multiminimize.conjugate_pr>`_ (:code:`conjugate_pr`)
-  
+
 * `Fletcher-Reeves Conjugate-Gradient <http://pygsl.sourceforge.net/api/pygsl.html#pygsl.multiminimize.conjugate_fr>`_ (:code:`conjugate_fr`)
-  
+
 * `The vector quasi-Newton BFGS method <http://pygsl.sourceforge.net/api/pygsl.html#pygsl.multiminimize.vector_bfgs>`_ (:code:`vector_bfgs`)
-  
+
 * `The vector quasi-Newton BFGS method (version 2) <http://pygsl.sourceforge.net/api/pygsl.html#pygsl.multiminimize.vector_bfgs2>`_ (:code:`vector_bfgs2`)
-  
+
 * `Steepest Descent <http://pygsl.sourceforge.net/api/pygsl.html#pygsl.multiminimize.steepest_descent>`_ (:code:`steepest_descent`)
 
 **Links** `SourceForge PyGSL <http://pygsl.sourceforge.net/>`_
@@ -91,21 +91,21 @@ manipulate and analyze neutron scattering and muon spectroscopy data.
 It has support for a number of minimizers, most of which are from GSL.
 
 * `BFGS <https://docs.mantidproject.org/nightly/fitting/fitminimizers/BFGS.html>`_ (:code:`BFGS`)
-  
+
 * `Conjugate gradient (Fletcher-Reeves) <https://docs.mantidproject.org/nightly/fitting/fitminimizers/FletcherReeves.html>`_ (:code:`Conjugate gradient (Fletcher-Reeves imp.)`)
 
 * `Conjugate gradient (Polak-Ribiere) <https://docs.mantidproject.org/nightly/fitting/fitminimizers/PolakRibiere.html>`_ (:code:`Conjugate gradient (Polak-Ribiere imp.)`)
 
 * `Damped GaussNewton <https://docs.mantidproject.org/nightly/fitting/fitminimizers/DampedGaussNewton.html>`_ (:code:`Damped GaussNewton`)
 
 * `Levenberg-Marquardt algorithm <https://docs.mantidproject.org/nightly/fitting/fitminimizers/LevenbergMarquardt.html>`_ (:code:`Levenberg-Marquardt`)
-  
+
 * `Levenberg-Marquardt MD <https://docs.mantidproject.org/nightly/fitting/fitminimizers/LevenbergMarquardtMD.html>`_ (:code:`Levenberg-MarquardtMD`) - An implementation of Levenberg-Marquardt intended for MD workspaces, where work is divided into chunks to achieve a greater efficiency for a large number of data points.
 
 * `Simplex <https://docs.mantidproject.org/nightly/fitting/fitminimizers/Simplex.html>`_ (:code:`simplex`)
 
 * `SteepestDescent <https://docs.mantidproject.org/nightly/fitting/fitminimizers/Simplex.html>`_ (:code:`SteepestDescent`)
-  
+
 * `Trust Region <https://docs.mantidproject.org/nightly/fitting/fitminimizers/TrustRegion.html>`_ (:code:`Trust Region`) - An implementation of one of the algorithms available in RALFit.
 
  **Links** `GitHub - Mantid <https://github.com/mantidproject/mantid>`_ `Mantid's Fitting Docs <https://docs.mantidproject.org/nightly/algorithms/Fit-v1.html>`_
@@ -115,7 +115,7 @@ It has support for a number of minimizers, most of which are from GSL.
    The external package Mantid must be installed to use these minimizers.
    See :ref:`InstallMantid`
 
- 
+
 Minuit
 ------
 
@@ -149,9 +149,30 @@ order derivatives are currently utilized in FitBenchmarking.
 SciPy
 -----
 
+`SciPy <https://www.scipy.org>`_ is the standard python package for mathematical
+software.  In particular, we use the `minimize <https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html>`_
+solver for general minimization problems from the optimization chapter the
+SciPy's library. Currently we only use the algorithms that do not require
+Hessian information as inputs.
+
+* `Nelder-Mead algorithm <https://docs.scipy.org/doc/scipy/reference/optimize.minimize-neldermead.html>`_ (:code:`Nelder-Mead`)
+* `Powell algorithm <https://docs.scipy.org/doc/scipy/reference/optimize.minimize-powell.html>`_ (:code:`Powell`)
+* `Conjugate gradient algorithm <https://docs.scipy.org/doc/scipy/reference/optimize.minimize-cg.html>`_ (:code:`CG`)
+* `BFGS algorithm <https://docs.scipy.org/doc/scipy/reference/optimize.minimize-bfgs.html>`_ (:code:`BFGS`)
+* `Newton-CG algorithm <https://docs.scipy.org/doc/scipy/reference/optimize.minimize-newtoncg.html>`_  (:code:`Newton-CG`)
+* `L-BFGS-B algorithm <https://docs.scipy.org/doc/scipy/reference/optimize.minimize-lbfgsb.html>`_ (:code:`L-BFGS-B`)
+* `Truncated Newton (TNC) algorithm <https://docs.scipy.org/doc/scipy/reference/optimize.minimize-tnc.html>`_ (:code:`TNC`)
+* `Sequential Least SQuares Programming <https://docs.scipy.org/doc/scipy/reference/optimize.minimize-slsqp.html>`_ (:code:`SLSQP`)
+
+**Links** `Github - SciPy minimize <https://github.com/scipy/scipy/blob/master/scipy/optimize/_minimize.py>`_
+
+SciPy LS
+--------
+
 `SciPy <https://www.scipy.org>`_ is the standard python package for mathematical
 software.  In particular, we use the `least_squares <https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.least_squares.html#scipy.optimize.least_squares>`_
-solver from the optimization chapter the SciPy's library.
+solver for Least-Squares minimization problems from the optimization chapter
+the SciPy's library.
 
 * Levenberg-Marquardt with supplied Jacobian (:code:`lm-scipy`)  - a wrapper around MINPACK
 * Levenberg-Marquardt with no Jacobian passed (:code:`lm-scipy-no-jac`)  - as above, but using MINPACK's approximate Jacobian

diff --git a/docs/source/contributors/guidelines.rst b/docs/source/contributors/guidelines.rst
@@ -1,8 +1,16 @@
 .. _guidelines:
 
-######################
-Contributor Guidelines
-######################
+################
+Coding Standards
+################
+
+All code submitted must meet certain standards, outlined below, before
+it can be merged into the master branch.  It is the contributor's
+job to ensure that the following is satisfied, and the reviewer's
+role to check that these guidelines have been followed.
+
+The workflow to be used for submitting new code/issues is described in
+:ref:`workflow`.
 
 =======
 Linting
@@ -17,15 +25,10 @@ for this.
 Documentation
 =============
 
-Any new code will be accepted only if there is relevent documentation for it in
-the corresponding pull request.
-
-===
-Git
-===
+Any new code will be accepted only if the documentation, written in sphinx and
+found in `docs/`, has been updated accordingly, and the docstrings in the code
+have been updated where neccessary.
 
-Branches should be created from master, and by convention follow the pattern
-"nnn_description" where nnn is the issue number.
 
 =======
 Testing

diff --git a/docs/source/contributors/index.rst b/docs/source/contributors/index.rst
@@ -12,6 +12,7 @@ Here you will find all you need in order to get started.
     :caption: Contents:
 
     guidelines
+    workflow
     structure
     Module Index <module_index/fitbenchmarking>
 
diff --git a/docs/source/contributors/workflow.rst b/docs/source/contributors/workflow.rst
@@ -0,0 +1,74 @@
+.. _workflow:
+
+############
+Git Workflow
+############
+
+
+======
+Issues
+======
+
+All new work should start with a
+`new GitHub issue <https://github.com/fitbenchmarking/fitbenchmarking/issues/new/choose>`_
+being filed.
+This should clearly explain what the change to the code will do.
+There are templates for *Bug report*, *Documentation*,
+*Feature request* and *Test* issues on GitHub, and you can also
+open a blank issue if none of these work.
+
+If issues help meet a piece of work agreed with our funders, it
+is linked to the appropriate `Milestone <https://github.com/fitbenchmarking/fitbenchmarking/milestones>`_ in GitHub.
+
+===============
+Adding new code
+===============
+
+The first step in adding new code is to create a branch, where the work
+will be done. Branches should be named according to the convention
+"<nnn>-description", where <nnn> is the issue number.
+
+Please ensure everything required in :ref:`guidelines` is included in
+the branch.
+
+When you think your new code is ready to be merged into the codebase,
+you should open a pull request.  The description should contain the
+words "Fixes #<nnn>", where <nnn> is the issue number; this will ensure
+the issue is closed when the code is merged into master.  At this point
+the automated tests will trigger, and you can see if the code passes on
+an independent system.
+
+Sometimes it is desirable to open a pull request when the code is not
+quite ready to be merged.  This is a good idea, for example, if you want
+to get an early opinion on a coding descision.  If this is the case, you
+should mark the pull request as a *draft* on GitHub.
+
+Once the work is ready to be reviewed, you may want to assign a reviewer,
+if you think someone would be well suited to review this change.  It is worth
+messaging them on, for example, Slack, as well as requesting their review on
+GitHub.
+
+================
+Release branches
+================
+
+Most work should be branched off, and merged into, master.
+
+The exception is when we are near a release, and then the contributor
+must make the decision of whether the code will enter this release, or
+wait for a future one.
+
+Branches named `release-*` are protected branches; code must be approved by
+a reviewer before being added to them, and automated tests will be run on
+pull requests to these branches.  If code is to be included in the release, it
+must be pulled into this branch and not master.
+
+Release branches should have the format `release-major.minor.x`, starting from
+`release-0.1.x`.  When the code is released, we will tag that commit with
+`release-0.1.0`.  Any hotfixes will increment `x` by one, and a new tag will
+be created accordingly.  If at some point we don't want to provide hot-fixes
+to a given minor release, then the corresponding release branch may be deleted.
+
+When a pull request is merged into a release branch, then the change should also
+be merged into master as soon a possible.  As long as the tests pass, and there
+are no merge conflicts, this can be done without a detailed review.
diff --git a/docs/source/extending/controllers.rst b/docs/source/extending/controllers.rst
@@ -54,8 +54,17 @@ In order to add a new controller, you will need to:
    Unless the new controller is more complicated than the currently available
    controllers, this can be done by following the example of the others.
 
+6. In the :ref:`Minimizers` page of the :ref:`how` page, update with the
+   new software and minimizers following the structure there. Note: make
+   sure that you use `<software_name>` in :ref:`Minimizers` so that the
+   software links in the HTML tables link correctly to the documentation.
+
+7. At the bottom of :doc:`main index page <../index>`, add the logo of the
+   of the software package in the `Currently Benchmarking` section.
+
 .. note::
-   For ease of maintenance, please add new controllers to a list of software in alphabetical order.
+   For ease of maintenance, please add new controllers to a list of
+   software in alphabetical order.
 
 
 The :class:`~fitbenchmarking.parsing.fitting_problem.FittingProblem` and :class:`~fitbenchmarking.jacobian.base_jacobian.Jacobian` classes

diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -1,3 +1,5 @@
+.. _mainindex:
+
 .. FitBenchmarking documentation master file, created by
    sphinx-quickstart on Wed Sep 11 09:17:28 2019.
    You can adapt this file completely to your liking, but it should at least

diff --git a/examples/all_softwares.ini b/examples/all_softwares.ini
@@ -8,6 +8,15 @@
 # entry for the software with a newline separated list of minimizers.
 # default is all available minimizers as follows:
 
+# bumps: available minimizers (amoeba, lm-bumps, newton, de, mp)
+#          for more information see
+#          https://bumps.readthedocs.io/en/latest/guide/optimizer.html
+#bumps: amoeba
+#       lm-bumps
+#       newton
+#       de
+#       mp
+
 # dfo: available minimimizers (dfogn, dfols)
 #        for more information see
 #        http://people.maths.ox.ac.uk/robertsl/dfogn/
@@ -64,28 +73,32 @@
 #        hybrid
 #        hybrid_reg
 
-# sasview: available minimizers (amoeba, lm-bumps, newton, de, mp)
-#          for more information see
-#          https://bumps.readthedocs.io/en/latest/guide/optimizer.html
-#sasview: amoeba
-#         lm-bumps
-#         newton
-#         de
-#         mp
-
-# scipy: available minimizers (lm-scipy, trf, dogbox)
-#        for more information see scipy.optimize.least_squares.html
+# scipy: available minimizers (Nelder-Mead, Powell, CG, BFGS, Newton-CG,
+#        L-BFGS-B, TNC, SLSQP)
+#        for more information see scipy.optimize.minimize.html
 #        https://docs.scipy.org/doc/scipy/reference/generated/
+scipy: Nelder-Mead
+       Powell
+       CG
+       BFGS
+       Newton-CG
+       L-BFGS-B
+       TNC
+       SLSQP
+
+# scipy_ls: available minimizers (lm-scipy-no-jac, lm-scipy, trf, dogbox)
+#           for more information see scipy.optimize.least_squares.html
+#           https://docs.scipy.org/doc/scipy/reference/generated/
 # Note: The minimizer "lm-scipy-no-jac" uses MINPACK's Jacobian evaluation
 #       which are significantly faster and gives different results than
-#       using the minimizer "lm-scipy" which uses problem.eval_j for the
+#       using the minimizer "lm-scipy" which uses problem.jac.eval for the
 #       Jacobian evaluation. We do not see significant speed changes or
 #       difference in the accuracy results when running trf or dogbox with
-#       or without problem.eval_j for the Jacobian evaluation
-#scipy: lm-scipy-no-jac
-#       lm-scipy
-#       trf
-#       dogbox
+#       or without problem.jac.eval for the Jacobian evaluation
+scipy_ls: lm-scipy-no-jac
+          lm-scipy
+          trf
+          dogbox
 
 
 ##############################################################################
@@ -99,13 +112,13 @@
 
 # software is used to select the fitting software to benchmark, this should be
 # a newline-separated list
-# default is dfo, minuit, sasview, and scipy
-software: dfo
+# default is bumps, dfo, gsl, mantid, minuit, ralfit, and scipy
+software: bumps
+          dfo
           gsl
           mantid
           minuit
           ralfit
-          sasview
           scipy
 
 # use_errors will switch between weighted and unweighted least squares