Parallel Differential Evolution based on pools

[pavelponomarev]

The Diferential Evolution algorithm modified for parallel execution. This is beneficial for computationally-expensive objective functions. MPI and joblib pools are also provided taken from emcee project.The API is modified to fix legacy problems and use particulars of population-based optimization algorithms in full. Solves #4864.

[dlax]

A few remarks following a quick review of your commits:

• when you introduce backwards incompatible changes, you need to add some deprecation warnings
• there is no test related to your changes, we need some
• there are some commits at the end that do not appear clearly related to parallelization of DE, these should be submitted as separate PR
• the commit history is quite messy, it would ease reviewing if you could rewrite the history by isolating changes into atomic commits and avoid changing things back and forth between commits

[pavelponomarev]

Yes,
Before I rewrite the code, please check the first and second commits. Is this a right way to include additional helper modules (pools) to the directory tree of the project? Is additional folder pools acceptable? Is the usage of the file pools/__init__.py correct?

[pv]

To the technical parts of adding the pools: . (i) having same name for the module and the classes causes problems. (ii) the pool modules should be private, ie., rename `Spool.py` -> `_spool.py` (iii) in addition to adding the pools to `__all__` in `__init__.py`, you need to import them, via `from ._spool import SPool`. (iv) nitpick -- the classes should be named `SPool`, `MPIPool`, `JLPool` (maybe `JoblibPool`) to follow proper camelcase . The bikeshedding question is whether `scipy.optimize.pools` is a good place for them. Perhaps not, because they could be useful also for other things than `scipy.optimize`. There are a some other alternatives such as putting them in `scipy.misc`, or, putting them in `scipy._lib` and importing from there to `scipy.optimize` top-level (and similarly in other modules using them).

[andyfaff]

I think that the introduction of pool code is going to complicate this PR. Presumably all the pool objects have a map method. A default pool map for testing parallelisation could be from the multiprocessing module. Users can supply their own pool objects instead if they want to.
It's not necessary to introduce these pools to scipy to continue work on this PR at this point.

[pavelponomarev]

@andyfaff , agree.
Then there should be a serial pool, which is a wrapper over normal serial execution of the objective function, and a parallel pool based on multiprocessing with minimal functionality. The proper place for these two pools is then in scipy.misc.

[rgommers]

To who looks at this PR: the main discussion on how the API for parallelization should look is in gh-4864. This PR cannot be merged until a decision is made there.

[pavelponomarev]

This PR is broken to several smaller PRs. Follow here #5141.

[rgommers]

Thanks @pavelponomarev