Tutorial AE MNIST added added (corrections 0)

docs/tutorials/AE_mnist.rst

@@ -0,0 +1,40 @@
+.. _AE_MNIST:
+
+Autoencoder on MNIST
+==========================================================
+
+Example for training a centered Autoencoder on the MNIST handwritten digit dataset with and without contractive penalty,
+dropout, ...
+
+It allows to reproduce the results from the publication `How to Center Deep Boltzmann Machines. Melchior et al. JMLR 2016. <http://jmlr.org/papers/v17/14-237.html>`_. Running the code as it is for example reproduces a single trial of the plot in Figure 9. (PCD-1) for $dd^b_s$.
+
+Theory
+***********
+
+If you are new on Autoencoders visit `Autoencoder tutorial <http://ufldl.stanford.edu/wiki/index.php/Autoencoders_and_Sparsity>`_ or watch the video course by Hugo Larochelle
+
+.. raw:: html
+
+    <div style="margin-top:10px;">
+      <iframe width="560" height="315" src="http://www.youtube.com/watch?v=FzS3tMl4Nsc" frameborder="0" allowfullscreen></iframe>
+    </div>
+
+Results
+***********
+
+The code_ given below produces the following output that is quite similar to the results produced by an RBM.
+
+
+
+See also `RBM_MNIST <RBM_MNIST.html#RBM_MNIST>`__.
+.. _code:
+
+
+Source code
+***********
+
+.. figure:: images/download_icon.png
+   :scale: 20 %
+   :target: https://github.com/MelJan/PyDeep/blob/master/examples/AE_mnist.py
+
+.. literalinclude:: ../../examples/AE_mnist.py

docs/tutorials/AE_natural_images.rst

@@ -13,7 +13,7 @@ If you are new on Autoencoders visit `Autoencoder tutorial <http://ufldl.stanfor
 .. raw:: html
 
     <div style="margin-top:10px;">
-      <iframe width="560" height="315" src="https://www.youtube.com/watch?v=FzS3tMl4Nsc" frameborder="0" allowfullscreen></iframe>
+      <iframe width="560" height="315" src="http://www.youtube.com/watch?v=FzS3tMl4Nsc" frameborder="0" allowfullscreen></iframe>
     </div>
 
 Results
@@ -78,11 +78,10 @@ We can also train the model on the unwhitened data leading to the following filt
    :align: center
 
 See also `GRBM_natural_images <GRBM_natural_images.html#GRBM_natural_images>`__,
-and `ICA_natural_images <ICA_natural_images.html#ICA_natural_images>`__..
+and `ICA_natural_images <ICA_natural_images.html#ICA_natural_images>`__.
 
 .. _code:
 
-.. _code:
 
 Source code
 ***********

docs/tutorials/ICA_2D_example.rst

@@ -70,6 +70,8 @@ The log-likelihood on all data is:
 
 For a real-world application see the `ICA_natural_images <ICA_natural_images.html#_ICA_Natural_Images>`__ example.
 
+.. _code:
+
 Source code
 ***********
 

docs/tutorials/ICA_natural_images.rst

@@ -67,7 +67,8 @@ Furthermore, we can plot the histogram of all filters over the frequencies in pi
    :align: center
 
 See also `GRBM_natural_images <GRBM_natural_images.html#GRBM_natural_images>`__.
-and `AE_natural_images <AE_natural_images.html#AE_natural_images>`__..
+and `AE_natural_images <AE_natural_images.html#AE_natural_images>`__.
+
 .. _code:
 
 Source code

docs/tutorials/RBM_MNIST_big.rst

@@ -67,6 +67,8 @@ depends on the training setup, whereas centering is rather robust to that.
 
 For real valued input see also `GRBM_natural_images <GRBM_natural_images.html#GRBM_natural_images>`__.
 
+.. _code:
+
 Source code
 ***********
 

examples/AE_natural_images.py

@@ -75,7 +75,7 @@
 batch_size = 10
 max_epochs = 20
 
-# Create model with sigmoid hidden units, linear output units, and squared error loss.
+# Create model with sigmoid hidden units, linear output units, and squared error.
 ae = aeModel.AutoEncoder(v1*v2,
                          h1*h2,
                          data = train_data,
@@ -84,7 +84,8 @@
                          cost_function = cost.SquaredError(),
                          initial_weights = 0.01,
                          initial_visible_bias = 0.0,
-                         initial_hidden_bias = -2.0, # Set initially the units to be inactive, speeds up learning a little bit
+                         initial_hidden_bias = -2.0,
+# Set initially the units to be inactive, speeds up learning a little bit
                          initial_visible_offsets = 0.0,
                          initial_hidden_offsets = 0.02,
                          dtype = numx.float64)
@@ -101,9 +102,10 @@
     train_data = numx.random.permutation(train_data)
 
     # Print reconstruction errors and sparseness for Training and test data
-    print epoch, ' \t\t', numx.mean(ae.reconstruction_error(train_data)), ' \t',\
-        numx.mean(ae.reconstruction_error(test_data)), ' \t', numx.mean(ae.encode(train_data)), ' \t',\
-        numx.mean(ae.encode(test_data))
+    print epoch, ' \t\t', numx.mean(ae.reconstruction_error(train_data)), \
+        ' \t', numx.mean(ae.reconstruction_error(test_data)),\
+        ' \t', numx.mean(ae.encode(train_data)), \
+        ' \t', numx.mean(ae.encode(test_data))
     for b in range(0,train_data.shape[0],batch_size):
 
         trainer.train(data = train_data[b:(b+batch_size),:],
@@ -115,24 +117,30 @@
                       reg_L1Norm=0.0,
                       reg_L2Norm=0.0,
                       corruptor=None,
-                      reg_sparseness = 2.0, # Rather strong sparsity regularization
+                      # Rather strong sparsity regularization
+                      reg_sparseness = 2.0,
                       desired_sparseness=0.001,
                       reg_contractive=0.0,
                       reg_slowness=0.0,
                       data_next=None,
-                      restrict_gradient=0.1, # The gradient restriction is important for fast learning, see also GRBMs
+# The gradient restriction is important for fast learning, see also GRBMs
+                      restrict_gradient=0.1,
                       restriction_norm='Cols')
 
 # Show filters/features
-filters = vis.tile_matrix_rows(ae.w, v1,v2,h1,h2, border_size = 1,normalized = True)
+filters = vis.tile_matrix_rows(ae.w, v1,v2,h1,h2, border_size = 1,
+                               normalized = True)
 vis.imshow_matrix(filters, 'Filter')
 
 # Show samples
-samples = vis.tile_matrix_rows(train_data[0:100].T, v1,v2,10,10, border_size = 1,normalized = True)
+samples = vis.tile_matrix_rows(train_data[0:100].T, v1,v2,10,10,
+                               border_size = 1,normalized = True)
 vis.imshow_matrix(samples, 'Data samples')
 
 # Show reconstruction
-samples = vis.tile_matrix_rows(ae.decode(ae.encode(train_data[0:100])).T, v1,v2,10,10, border_size = 1,normalized = True)
+samples = vis.tile_matrix_rows(ae.decode(ae.encode(train_data[0:100])).T,
+                               v1,v2,10,10, border_size = 1,
+                               normalized = True)
 vis.imshow_matrix(samples, 'Reconstructed samples')
 
 # Get the optimal gabor wavelet frequency and angle for the filters