Merge pull request #9 from epfl-lts2/acceleration

Add acceleration schemes as a separate class hierarchy so that they can be mixed and matched with solvers.

.travis.yml

@@ -6,9 +6,9 @@ language: python
 
 python:
   - "2.7"
-  - "3.3"
   - "3.4"
   - "3.5"
+  - "3.6"
 
 addons:
   apt:
@@ -20,6 +20,7 @@ addons:
       - texlive-latex-recommended
       - texlive-latex-extra
       - texlive-fonts-recommended
+      - latexmk
 
 install:
   - pip install --upgrade pip

CONTRIBUTING.rst

@@ -0,0 +1,112 @@
+============
+Contributing
+============
+
+Contributions are welcome, and they are greatly appreciated! Every
+little bit helps, and credit will always be given. 
+
+You can contribute in many ways:
+
+Types of Contributions
+----------------------
+
+Report Bugs
+~~~~~~~~~~~
+
+Report bugs at https://github.com/epfl-lts2/pyunlocbox/issues.
+
+If you are reporting a bug, please include:
+
+* Your operating system name and version.
+* Any details about your local setup that might be helpful in troubleshooting.
+* Detailed steps to reproduce the bug.
+
+Fix Bugs
+~~~~~~~~
+
+Look through the GitHub issues for bugs. Anything tagged with "bug"
+is open to whoever wants to fix it.
+
+Implement Features
+~~~~~~~~~~~~~~~~~~
+
+Look through the GitHub issues for features. Anything tagged with "feature"
+is open to whoever wants to implement it.
+
+Write Documentation
+~~~~~~~~~~~~~~~~~~~
+
+PyGSP could always use more documentation, whether as part of the 
+official PyGSP docs, in docstrings, or even on the web in blog posts,
+articles, and such.
+
+Submit Feedback
+~~~~~~~~~~~~~~~
+
+The best way to send feedback is to file an issue at https://github.com/epfl-lts2/pyunlocbox/issues.
+
+If you are proposing a feature:
+
+* Explain in detail how it would work.
+* Keep the scope as narrow as possible, to make it easier to implement.
+* Remember that this is a volunteer-driven project, and that contributions
+  are welcome :)
+
+Get Started!
+------------
+
+Ready to contribute? Here's how to set up `pyunlocbox` for local development.
+
+1. Fork the `pyunlocbox` repo on GitHub.
+2. Clone your fork locally::
+
+    $ git clone git@github.com:your_name_here/pyunlocbox.git
+
+3. Install your local copy into a virtualenv. Assuming you have `virtualenvwrapper <https://virtualenvwrapper.readthedocs.org/en/latest/>`_ installed, this is how you set up your fork for local development::
+
+    $ mkvirtualenv pyunlocbox
+    $ cd pyunlocbox/
+    $ python setup.py develop
+
+Note: alternatively, the third step could be replaced by::
+
+    $ pip install -e .
+
+4. Create a branch for local development::
+
+    $ git checkout -b name-of-your-bugfix-or-feature
+   
+   Now you can make your changes locally.
+
+5. For each new feature/bugfix you develop, please create a respective test case under pyunlocbox/pyunlocbox/tests/. It might be that you can just add a new test function to the existing test scripts, or that it makes more sense to create a new test script altogether. In the latter case, make sure to add a reference to the new test script test_all.py .
+
+6. When you're done making changes, check that your changes pass the tests, and, ideally flake8, by running the following commands from your local pyunlocbox directory::
+
+    $ make test
+    $ make lint
+   
+   Check the generated coverage report at pyunlocbox/htmlcov/index.html to make sure that the tests fully (or mostly) cover the changes you’ve introduced to the repository. If that’s not the case, add more test cases and repeat step 6.
+
+7. Commit your changes and push your branch to GitHub::
+
+    $ git add *
+    $ git commit -m "Your detailed description of your changes."
+    $ git push --set-upstream origin name-of-your-branch
+
+8. Submit a pull request through the GitHub website, but make sure to read the pull request guidelines below.
+
+Pull Request Guidelines
+-----------------------
+
+Before you submit a pull request, check that it meets these guidelines:
+
+1. The pull request should include tests.
+
+2. If the pull request adds functionality, the docs should be updated. Put
+   your new functionality into a function with a docstring, and add the
+   feature to the list in README.rst.
+
+3. The pull request should work for Python 2.7, 3.4, 3.5 and 3.6. Check 
+   https://travis-ci.org/epfl-lts2/pygsp/pull_requests
+   and make sure that the tests pass for all supported Python versions.
+

README.rst

@@ -6,18 +6,20 @@ PyUNLocBoX is a convex optimization toolbox using proximal splitting methods
 implemented in Python. It is a free software distributed under the BSD license
 and is a port of the Matlab UNLocBoX toolbox.
 
-.. image:: https://img.shields.io/travis/epfl-lts2/pyunlocbox.svg
-   :target: https://travis-ci.org/epfl-lts2/pyunlocbox
+.. only:: html
 
-.. image:: https://img.shields.io/coveralls/epfl-lts2/pyunlocbox.svg
-   :target: https://coveralls.io/github/epfl-lts2/pyunlocbox
+  .. image:: https://img.shields.io/travis/epfl-lts2/pyunlocbox.svg
+     :target: https://travis-ci.org/epfl-lts2/pyunlocbox
 
-.. image:: https://img.shields.io/pypi/v/pyunlocbox.svg
-   :target: https://pypi.python.org/pypi/pyunlocbox
+  .. image:: https://img.shields.io/coveralls/epfl-lts2/pyunlocbox.svg
+     :target: https://coveralls.io/github/epfl-lts2/pyunlocbox
 
-.. image:: https://img.shields.io/pypi/l/pyunlocbox.svg
+  .. image:: https://img.shields.io/pypi/v/pyunlocbox.svg
+     :target: https://pypi.python.org/pypi/pyunlocbox
 
-.. image:: https://img.shields.io/pypi/pyversions/pyunlocbox.svg
+  .. image:: https://img.shields.io/pypi/l/pyunlocbox.svg
+
+  .. image:: https://img.shields.io/pypi/pyversions/pyunlocbox.svg
 
 * Development : https://github.com/epfl-lts2/pyunlocbox
 * Documentation : https://pyunlocbox.readthedocs.io
@@ -42,6 +44,12 @@ Features
   * TV-norm
   * Projection on the L2-ball
 
+* Acceleration schemes
+
+  * FISTA acceleration scheme
+  * Backtracking based on a quadratic approximation of the objective
+  * Regularized nonlinear acceleration (RNA)
+
 Following is a typical usage example who solves an optimization problem
 composed by the sum of two convex functions. The functions and solver objects
 are first instantiated with the desired parameters. The problem is then solved
@@ -52,16 +60,16 @@ by a call to the solving function.
 >>> f2 = pyunlocbox.functions.dummy()
 >>> solver = pyunlocbox.solvers.forward_backward()
 >>> ret = pyunlocbox.solvers.solve([f1, f2], [0., 0, 0, 0], solver, atol=1e-5)
-Solution found after 10 iterations:
-    objective function f(sol) = 7.460428e-09
+Solution found after 9 iterations:
+    objective function f(sol) = 6.714385e-08
     stopping criterion: ATOL
 >>> ret['sol']
-array([ 3.99996922,  4.99996153,  5.99995383,  6.99994614])
+array([ 3.99990766,  4.99988458,  5.99986149,  6.99983841])
 
 Installation
 ------------
 
-PyUnLocBox is continuously tested on Python 2.7, 3.3, 3.4 and 3.5.
+PyUnLocBox is continuously tested on Python 2.7, 3.4, 3.5, and 3.6.
 
 System-wide installation::
 

doc/reference/acceleration.rst

@@ -0,0 +1,51 @@
+============================
+Acceleration class hierarchy
+============================
+
+Acceleration scheme object interface
+------------------------------------
+
+.. autoclass:: pyunlocbox.acceleration.accel
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+Dummy acceleration scheme
+-------------------------
+
+.. autoclass:: pyunlocbox.acceleration.dummy
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+Backtracking from quadratic approximation
+-----------------------------------------
+
+.. autoclass:: pyunlocbox.acceleration.backtracking
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+FISTA acceleration scheme
+-------------------------
+
+.. autoclass:: pyunlocbox.acceleration.fista
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+FISTA acceleration scheme with backtracking
+-------------------------------------------
+
+.. autoclass:: pyunlocbox.acceleration.fista_backtracking
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+Regularized Nonlinear Acceleration (RNA)
+----------------------------------------
+
+.. autoclass:: pyunlocbox.acceleration.regularized_nonlinear
+    :members:
+    :undoc-members:
+    :show-inheritance:

doc/reference/index.rst

@@ -36,6 +36,20 @@ Solvers module
 .. inheritance-diagram:: pyunlocbox.solvers
     :parts: 2
 
+Acceleration module
+-------------------
+
+.. toctree::
+    :hidden:
+
+    acceleration
+
+.. automodule:: pyunlocbox.acceleration
+
+.. inheritance-diagram:: pyunlocbox.acceleration
+    :parts: 2
+
+
 Operators module
 ----------------
 

doc/reference/solvers.rst

@@ -10,6 +10,14 @@ Solver object interface
     :undoc-members:
     :show-inheritance:
 
+Gradient descent algorithm
+--------------------------
+
+.. autoclass:: pyunlocbox.solvers.gradient_descent
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
 Forward-backward proximal splitting algorithm
 ---------------------------------------------
 

doc/references.bib

@@ -54,6 +54,16 @@ @article{beck2009fastTV
   publisher={IEEE}
 }
 
+@incollection{combettes:2011iq,
+author = {Combettes, Patrick L and Pesquet, Jean-Christophe},
+title = {{Proximal Splitting Methods in Signal Processing}},
+booktitle = {Fixed-Point Algorithms for Inverse Problems in Science and Engineering},
+year = {2011},
+pages = {185--212},
+publisher = {Springer New York},
+address = {New York, NY}
+}
+
 ----  Tutorials  ----
 
 @article{candes2007CSperfect,
@@ -66,3 +76,18 @@ @article{candes2007CSperfect
   year={2007},
   publisher={IOP Publishing}
 }
+
+---- Acceleration ----
+
+@article{scieur2016,
+author = {Scieur, Damien and dAspremont, Alexandre and Bach, Francis},
+title = {{Regularized Nonlinear Acceleration}},
+journal = {arxiv.org},
+year = {2016},
+eprint = {1606.04133},
+eprinttype = {arxiv},
+eprintclass = {math.OC},
+pages = {arXiv:1606.04133},
+month = jun
+}
+

doc/tutorials/compressed_sensing_douglas_rachford.rst

@@ -19,14 +19,14 @@ reconstruction. See :cite:`candes2007CSperfect` for details.
 .. plot::
    :context: reset
 
-   >>> n = 5000
-   >>> S = 100
+   >>> n = 900
+   >>> S = 45
    >>> import numpy as np
    >>> m = int(np.ceil(S * np.log(n)))
    >>> print('Number of measurements: {}'.format(m))
-   Number of measurements: 852
+   Number of measurements: 307
    >>> print('Compression ratio: {:3.2f}'.format(float(n) / m))
-   Compression ratio: 5.87
+   Compression ratio: 2.93
 
 We generate a random measurement matrix `A`:
 
@@ -101,8 +101,8 @@ follows:
 
    >>> x0 = np.zeros(n)
    >>> ret = solvers.solve([f1, f2], x0, solver, rtol=1e-4, maxit=300)
-   Solution found after 56 iterations:
-       objective function f(sol) = 7.590460e+00
+   Solution found after 43 iterations:
+       objective function f(sol) = 5.607407e+00
        stopping criterion: RTOL
 
 Let's display the results:

doc/tutorials/compressed_sensing_forward_backward.rst

@@ -119,7 +119,7 @@ instantiated as follows:
 
    >>> step = 0.5 / np.linalg.norm(A, ord=2)**2
    >>> from pyunlocbox import solvers
-   >>> solver = solvers.forward_backward(method='FISTA', step=step)
+   >>> solver = solvers.forward_backward(step=step)
 
 .. note:: A complete description of the constructor parameters and default
     values is given by the solver object
@@ -134,8 +134,8 @@ follows:
 
    >>> x0 = np.zeros(n)
    >>> ret = solvers.solve([f1, f2], x0, solver, rtol=1e-4, maxit=300)
-   Solution found after 152 iterations:
-       objective function f(sol) = 7.668195e+00
+   Solution found after 151 iterations:
+       objective function f(sol) = 7.668167e+00
        stopping criterion: RTOL
 
 .. note:: A complete description of the parameters, their default values and

doc/tutorials/reconstruction.rst

@@ -82,7 +82,7 @@ with
    :context:
 
    >>> from pyunlocbox import solvers
-   >>> solver = solvers.forward_backward(method='FISTA', step=0.5/tau)
+   >>> solver = solvers.forward_backward(step=0.5/tau)
 
 and the problem solved with
 
@@ -91,8 +91,8 @@ and the problem solved with
 
    >>> x0 = np.array(im_masked)  # Make a copy to preserve im_masked.
    >>> ret = solvers.solve([f1, f2], x0, solver, maxit=100)
-   Solution found after 94 iterations:
-       objective function f(sol) = 4.268147e+03
+   Solution found after 93 iterations:
+       objective function f(sol) = 4.268861e+03
        stopping criterion: RTOL
 
 Let's display the results: