Auto notebooks from examples with shpinx-gallery

.mailmap

@@ -1,3 +1,4 @@
 Nicolas Courty  <ncourty@irisa.fr> Nicolas Courty <Nico@MacBook-Pro-de-Nicolas.local>
 Nicolas Courty  <ncourty@irisa.fr> ncourty <ncourty@irisa.fr>
+Nicolas Courty  <ncourty@irisa.fr>  Nicolas Courty <Nico@pc-mna-08.univ-ubs.fr>
 Léo Gautheron <leo_g_autheron@hotmail.fr>  Leo gautheron <gautheron@iv-cm-359.creatis.insa-lyon.fr>

Makefile

@@ -15,6 +15,9 @@ help :
 build :
 	$(PYTHON) setup.py build
 
+buildext :
+	$(PYTHON) setup.py build_ext --inplace
+
 install :
 	$(PYTHON) setup.py install --user
 

README.md

@@ -112,17 +112,21 @@ The examples folder contain several examples and use case for the library. The f
 
 Here is a list of the Python notebooks available [here](https://github.com/rflamary/POT/blob/master/notebooks/) if you want a quick look:
 
-* [1D optimal transport](https://github.com/rflamary/POT/blob/master/notebooks/Demo_1D_OT.ipynb)
-* [OT Ground Loss](https://github.com/rflamary/POT/blob/master/notebooks/Demo_Ground_Loss.ipynb)
-* [Multiple EMD computation](https://github.com/rflamary/POT/blob/master/notebooks/Demo_Compute_EMD.ipynb)
-* [2D optimal transport on empirical distributions](https://github.com/rflamary/POT/blob/master/notebooks/Demo_2D_OT_samples.ipynb)
-* [1D Wasserstein barycenter](https://github.com/rflamary/POT/blob/master/notebooks/Demo_1D_barycenter.ipynb)
-* [OT with user provided regularization](https://github.com/rflamary/POT/blob/master/notebooks/Demo_Optim_OTreg.ipynb)
-* [Domain adaptation with optimal transport](https://github.com/rflamary/POT/blob/master/notebooks/Demo_2D_OT_DomainAdaptation.ipynb)
-* [Color transfer in images](https://github.com/rflamary/POT/blob/master/notebooks/Demo_Image_ColorAdaptation.ipynb)
-* [OT mapping estimation for domain adaptation](https://github.com/rflamary/POT/blob/master/notebooks/Demo_2D_OTmapping_DomainAdaptation.ipynb)
-* [OT mapping estimation for color transfer in images](https://github.com/rflamary/POT/blob/master/notebooks/Demo_Image_ColorAdaptation_mapping.ipynb)
-* [Wasserstein Discriminant Analysis](https://github.com/rflamary/POT/blob/master/notebooks/Demo_Wasserstein_Discriminant_Analysis.ipynb)
+* [1D optimal transport](https://github.com/rflamary/POT/blob/master/notebooks/plot_OT_1D.ipynb)
+* [OT Ground Loss](https://github.com/rflamary/POT/blob/master/notebooks/plot_OT_L1_vs_L2.ipynb)
+* [Multiple EMD computation](https://github.com/rflamary/POT/blob/master/notebooks/plot_compute_emd.ipynb)
+* [2D optimal transport on empirical distributions](https://github.com/rflamary/POT/blob/master/notebooks/plot_OT_2D_samples.ipynb)
+* [1D Wasserstein barycenter](https://github.com/rflamary/POT/blob/master/notebooks/plot_barycenter_1D.ipynb)
+* [OT with user provided regularization](https://github.com/rflamary/POT/blob/master/notebooks/plot_optim_OTreg.ipynb)
+* [Domain adaptation with optimal transport](https://github.com/rflamary/POT/blob/master/notebooks/plot_otda_d2.ipynb)
+* [Color transfer in images](https://github.com/rflamary/POT/blob/master/notebooks/plot_otda_color_images.ipynb)
+* [OT mapping estimation for domain adaptation](https://github.com/rflamary/POT/blob/master/notebooks/plot_otda_mapping.ipynb)
+* [OT mapping estimation for color transfer in images](https://github.com/rflamary/POT/blob/master/notebooks/plot_otda_mapping_colors_images.ipynb)
+* [Wasserstein Discriminant Analysis](https://github.com/rflamary/POT/blob/master/notebooks/plot_WDA.ipynb)
+* [Gromov Wasserstein](https://github.com/rflamary/POT/blob/master/notebooks/plot_gromov.ipynb)
+* [Gromov Wasserstein Barycenter](https://github.com/rflamary/POT/blob/master/notebooks/plot_gromov_barycenter.ipynb)
+
+
 
 You can also see the notebooks with [Jupyter nbviewer](https://nbviewer.jupyter.org/github/rflamary/POT/tree/master/notebooks/).
 

RELEASES.md

@@ -0,0 +1,88 @@
+# POT Releases
+
+## 0.4 Community edition
+*15 Sep 2017*
+
+This release contains a lot of contribution from new contributors.
+
+
+#### Features
+
+* Automatic notebooks and doc update (PR #27)
+* Add gromov Wasserstein solver and Gromov Barycenters (PR #23)
+* emd and emd2 can now return dual variables and have max_iter (PR #29 and PR #25) 
+* New domain adaptation classes compatible with scikit-learn (PR #22)
+* Proper tests with pytest on travis (PR #19)
+* PEP 8 tests (PR #13)
+
+#### Closed issues
+
+* emd convergence problem du to fixed max iterations (#24) 
+* Semi supervised DA error (#26)
+
+## 0.3.1
+*11 Jul 2017*
+
+* Correct bug in emd on windows
+
+## 0.3 Summer release
+*7 Jul 2017*
+
+* emd* and sinkhorn* are now performed in parallel for multiple target distributions
+* emd and sinkhorn are for OT matrix computation
+* emd2 and sinkhorn2 are for OT loss computation
+* new notebooks for emd computation and Wasserstein Discriminant Analysis
+* relocate notebooks
+* update documentation
+* clean_zeros(a,b,M) for removimg zeros in sparse distributions
+* GPU implementations for sinkhorn and group lasso regularization
+
+
+## V0.2 
+*7 Apr 2017*
+
+* New dimensionality reduction method (WDA)
+* Efficient method emd2 returns only tarnsport (in paralell if several histograms given)
+
+
+
+## V0.1.11 New years resolution
+*5 Jan 2017*
+
+* Add sphinx gallery for better documentation
+* Small efficiency tweak in sinkhorn
+* Add simple tic() toc() functions for timing
+
+
+## V0.1.10 
+*7 Nov 2016*
+* numerical stabilization for sinkhorn (log domain and epsilon scaling)
+
+## V0.1.9 DA classes and mapping
+*4 Nov 2016*
+
+* Update classes and examples for domain adaptation
+* Joint OT matrix and mapping estimation
+
+## V0.1.7
+*31 Oct 2016*
+
+* Original Domain adaptation classes
+
+
+
+## PyPI version 0.1.3
+
+* pipy works
+
+## First pre-release
+*28 Oct 2016*
+
+It provides the following solvers:
+* OT solver for the linear program/ Earth Movers Distance.
+* Entropic regularization OT solver  with Sinkhorn Knopp Algorithm.
+* Bregman projections for Wasserstein barycenter [3] and unmixing.
+* Optimal transport for domain adaptation with group lasso regularization
+* Conditional gradient and Generalized conditional gradient for regularized OT.
+
+Some demonstrations (both in Python and Jupyter Notebook format) are available in the examples folder.

docs/cache_nbrun

@@ -0,0 +1 @@
+{"plot_otda_mapping_colors_images.ipynb": "4f0587a00a3c082799a75a0ed36e9ce1", "plot_optim_OTreg.ipynb": "71d3c106b3f395a6b1001078a6ca6f8d", "plot_otda_color_images.ipynb": "d047d635f4987c81072383241590e21f", "plot_WDA.ipynb": "27f8de4c6d7db46497076523673eedfb", "plot_OT_L1_vs_L2.ipynb": "e15219bf651a7e39e7c5c3934069894c", "plot_barycenter_1D.ipynb": "6fd8167f98816dc832fe0c58b1d5527b", "plot_otda_classes.ipynb": "44bb8cd93317b5d342cd62e26d9bbe60", "plot_otda_d2.ipynb": "8ac4fd2ff899df0858ce1e5fead37f33", "plot_otda_mapping.ipynb": "d335a15af828aaa3439a1c67570d79d6", "plot_gromov.ipynb": "9d0893ec68851f200d0ca806bcbe847f", "plot_compute_emd.ipynb": "bd95981189df6adcb113d9b360ead734", "plot_OT_1D.ipynb": "e44c83f6112388ae18657cb0ad76d0e9", "plot_gromov_barycenter.ipynb": "a4d9636685394ceb13f26cdc613b9b5b", "plot_otda_semi_supervised.ipynb": "0261d339a692e339e15d3634488905cc", "plot_OT_2D_samples.ipynb": "3f125714daa35ff3cfe5dae1f71265c4"}

docs/nb_build

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+
+# remove comment 
+sed -i "s/#'sphinx\_gallery/'sphinx\_gallery/" source/conf.py
+sed -i "s/sys.modules.update/#sys.modules.update/" source/conf.py
+
+make html
+
+# put comment again 
+sed -i "s/'sphinx\_gallery/#'sphinx\_gallery/" source/conf.py
+sed -i "s/#sys.modules.update/sys.modules.update/" source/conf.py
+
+#rsync --out-format="%n" --update  source/auto_examples/*.ipynb ../notebooks2
+./nb_run_conv

docs/nb_run_conv

@@ -0,0 +1,82 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+
+Convert sphinx gallery notebook from empty to image filled
+
+Created on Fri Sep  1 16:43:45 2017
+
+@author: rflamary
+"""
+
+import sys
+import json
+import glob
+import hashlib
+import subprocess
+
+import os
+
+cache_file='cache_nbrun'
+
+path_doc='source/auto_examples/'
+path_nb='../notebooks/'
+
+def load_json(fname):
+    try:
+        f=open(fname)
+        nb=json.load(f)
+        f.close()
+    except (OSError, IOError) :
+        nb={}
+    return nb
+
+def save_json(fname,nb):
+    f=open(fname,'w')
+    f.write(json.dumps(nb))
+    f.close()
+
+
+def md5(fname):
+    hash_md5 = hashlib.md5()
+    with open(fname, "rb") as f:
+        for chunk in iter(lambda: f.read(4096), b""):
+            hash_md5.update(chunk)
+    return hash_md5.hexdigest()
+
+def to_update(fname,cache):
+    if fname in cache:
+        if md5(path_doc+fname)==cache[fname]:
+            res=False
+        else:
+            res=True
+    else:
+        res=True
+
+    return res
+
+def update(fname,cache):
+
+    # jupyter nbconvert --to notebook --execute mynotebook.ipynb --output targte
+    subprocess.check_call(['cp',path_doc+fname,path_nb])
+    print(' '.join(['jupyter','nbconvert','--to','notebook','--ExecutePreprocessor.timeout=600','--execute',path_nb+fname,'--inplace']))
+    subprocess.check_call(['jupyter','nbconvert','--to','notebook','--ExecutePreprocessor.timeout=600','--execute',path_nb+fname,'--inplace'])
+    cache[fname]=md5(path_doc+fname)
+
+
+
+cache=load_json(cache_file)
+
+lst_file=glob.glob(path_doc+'*.ipynb')
+
+lst_file=[os.path.basename(name) for name in lst_file]
+
+for fname in lst_file:
+    if to_update(fname,cache):
+        print('Updating file: {}'.format(fname))
+        update(fname,cache)
+        save_json(cache_file,cache)
+
+
+
+