General improvements.

- VAE is now implemented as shown in https://keras.io/examples/variational_autoencoder/ - Improved documentation. - Change dot_size in ml4chem/data/visualization.py. - Reconstruction loss VAE is now multiplied by input dimension.
muammar · Nov 1, 2019 · e026b11 · e026b11
1 parent 94fe182
commit e026b11
Show file tree

Hide file tree

Showing 7 changed files with 109 additions and 53 deletions.
diff --git a/README.md b/README.md
@@ -7,6 +7,7 @@
 [![Build Status](https://travis-ci.com/muammar/ml4chem.svg?branch=master)](https://travis-ci.com/muammar/ml4chem)
 [![License](https://img.shields.io/badge/license-BSD-green)](https://github.com/muammar/ml4chem/blob/master/LICENSE)
 [![Downloads](https://img.shields.io/github/downloads/muammar/ml4chem/total.svg?maxAge=2592000?style=flat-square)](https://github.com/muammar/ml4chem/releases)
+![PyPI - Downloads](https://img.shields.io/pypi/dm/ml4chem)
 [![GitHub release](https://img.shields.io/github/release/muammar/ml4chem.svg)](https://github.com/muammar/ml4chem/releases/latest)
 
 
@@ -46,9 +47,10 @@ index](https://ml4chem.dev/genindex.html) to get more information about
 different classes and functions of ML4Chem.
 
 
-## Dask dashboard
+## Visualizations
 ![](https://raw.githubusercontent.com/muammar/ml4chem/master/docs/source/_static/dask_dashboard.png)
 
+
 Note: This package is under development.
 
 ## Copyright

diff --git a/docs/source/_static/tsne_visual.html → docs/source/_static/pca_visual.html b/docs/source/_static/tsne_visual.html → docs/source/_static/pca_visual.html
diff --git a/docs/source/data.rst b/docs/source/data.rst
@@ -26,7 +26,7 @@ Its usage is very simple::
 
     images = Trajectory("images.traj")
     data_handler = DataSet(images, purpose="training")
-    traing_set, targets = data_handler.get_images(purpose="training")
+    traing_set, targets = data_handler.get_data(purpose="training")
 
 In the example above, an ASE trajectory file is loaded into memory and passed
 as an argument to instantiate the ``DataSet`` class with
@@ -50,15 +50,15 @@ An example is shown below::
 
     from ml4chem.data.visualization import plot_atomic_features
     fig = plot_atomic_features("latent_space.db", 
-                               method="tsne", 
+                               method="pca", 
                                dimensions=3, 
                                backend="plotly")
     fig.write_html("latent_example.html")
 
 This will produce an interactive plot with plotly where dimensionality was
-reduced using T-SNE, and an html with the name `latent_example.html` is
+reduced using PCA, and an html with the name `latent_example.html` is
 created.
 
 .. raw:: html
-   :file: _static/tsne_visual.html
+   :file: _static/pca_visual.html
 
diff --git a/docs/source/models.rst b/docs/source/models.rst
@@ -143,34 +143,69 @@ Models
 Neural Networks
 ----------------
 Neural Network (NN) are models inspired on how the human brain works. They
-consist of a set of hidden-layers with some nodes. The most simple NN
-architecture is the *fully-connected* case in which each neuron is connected
-to every neuron in the previous/next layer, and each connection has its own
+consist of a set of hidden-layers with some nodes (neurons). The most simple NN
+architecture is the *fully-connected* case in which each neuron is inter-connected
+to every other neuron in the previous/next layer, and each connection has its own
 weight. When an activation function is applied to the output of a
-hidden-layer, the NN is able to learn from non-linear data. 
+neuron, the NN is able to learn non-linearity aspects from the data.
+
+In ML4Chem, a neural network can be instantiated as shown below:
 
 :: 
 
     from ml4chem.models.neuralnetwork import NeuralNetwork
 
     n = 10
     activation = "relu"
-    model = NeuralNetwork(hiddenlayers=(n, n), activation=activation)
+    nn = NeuralNetwork(hiddenlayers=(n, n), activation=activation)
+    nn.prepare_model()
 
-In the example above, we are building a NN using the
+Here, weare building a NN with the
 :class:`ml4chem.models.neuralnetwork.NeuralNetwork` class with two
-hidden-layers of 10 neurons each, and a ReLu activation function.
+hidden-layers composed 10 neurons each, and a ReLu activation function.
 
 Autoencoders
 -------------
-Something here
+Autoencoders (AE) are NN architectures that able to extract features from
+data in an unsupervised learning manner. AE learns how to encode information
+because of a hidden-layer that serves as an informational bottleneck as shown
+in the figure below. In addition, this latent code is used by the decoder to
+reconstruct the input data.
+
+.. image:: https://en.wikipedia.org/wiki/Autoencoder#/media/File:Autoencoder_schema.png
+
+
+:: 
+
+    from ml4chem.models.autoencoders import AutoEncoder
+
+    hiddenlayers = {"encoder": (20, 10, 4), "decoder": (4, 10, 20)}
+    activation = "tanh"
+    autoencoder = AutoEncoder(hiddenlayers=hiddenlayers, activation=activation)
+    data_handler.get_unique_element_symbols(images, purpose=purpose)
+    autoencoder.prepare_model(input_dimension, output_dimension, data=data_handler)
+
+
+ML4Chem also provides access to variational autoencoders. It just suffices to
+change the snippet above as follows:
+
+:: 
+
+    from ml4chem.models.autoencoders import VAE
+
+    hiddenlayers = {"encoder": (20, 10, 4), "decoder": (4, 10, 20)}
+    activation = "tanh"
+    autoencoder = VAE(hiddenlayers=hiddenlayers, activation=activation)
+    data_handler.get_unique_element_symbols(images, purpose=purpose)
+    autoencoder.prepare_model(input_dimension, output_dimension, data=data_handler)
+
 
 Kernel Ridge Regression
 ------------------------
 Kernel Ridge Regression (KRR) is a type of support vector machine model that
-combines Ridge Regression with a kernel trick. In ML4Chem, this method is
-implemeted by Rupp in Ref. [Rupp2015]_. Below there is a description of this
-implementation:
+combines Ridge Regression with the kernel trick. In ML4Chem, this method is
+implemeted as described by Rupp in Ref. [Rupp2015]_. Below there is a
+description of this implementation:
 
 #. Molecules are featurized.  
 #. A kernel function :math:`k(x, y)` is applied to all possible pairs of
@@ -185,7 +220,6 @@ Gaussian Process Regression
 Gaussian Process Regression (GP) is similar to KRR with the addition of the
 uncertainty of each prediction.
 
-
 **References:**
 
 .. [Behler2007] Behler, J. & Parrinello, M. Generalized Neural-Network Representation of High-Dimensional Potential-Energy Surfaces. Phys. Rev. Lett. 98, 146401 (2007).

diff --git a/ml4chem/data/visualization.py b/ml4chem/data/visualization.py
@@ -108,14 +108,14 @@ def read_log(logfile, metric="loss", refresh=None):
 
         if initiliazed is False:
             if metric == "loss":
-                fig, = plt.plot(epochs, loss, label="loss")
+                (fig,) = plt.plot(epochs, loss, label="loss")
 
             elif metric == "rmse":
-                fig, = plt.plot(epochs, rmse, label="rmse")
+                (fig,) = plt.plot(epochs, rmse, label="rmse")
 
             else:
-                fig, = plt.plot(epochs, loss, label="loss")
-                fig, = plt.plot(epochs, rmse, label="rmse")
+                (fig,) = plt.plot(epochs, loss, label="loss")
+                (fig,) = plt.plot(epochs, rmse, label="rmse")
         else:
             if metric == "loss":
                 fig.set_data(epochs, loss)
@@ -162,25 +162,25 @@ def read_log(logfile, metric="loss", refresh=None):
                     pass
 
         if metric == "loss":
-            fig, = plt.plot(epochs, loss, label="loss")
+            (fig,) = plt.plot(epochs, loss, label="loss")
 
         elif metric == "rmse":
-            fig, = plt.plot(epochs, rmse, label="rmse")
+            (fig,) = plt.plot(epochs, rmse, label="rmse")
 
         else:
-            fig, = plt.plot(epochs, loss, label="loss")
-            fig, = plt.plot(epochs, rmse, label="rmse")
+            (fig,) = plt.plot(epochs, loss, label="loss")
+            (fig,) = plt.plot(epochs, rmse, label="rmse")
 
         plt.show(block=True)
 
 
-def plot_atomic_features(latent_space, method="PCA", dimensions=2, backend="seaborn"):
+def plot_atomic_features(latent_space, method="PCA", dimensions=3, backend="seaborn"):
     """Plot high dimensional atomic feature vectors
 
     This function can take a feature space dictionary, or a database file
     and plot the atomic features using PCA or t-SNE.
 
-    $ mlchem --plot tsne --file path.db
+    $ ml4chem --plot tsne --file path.db
 
     Parameters
     ----------
@@ -198,7 +198,7 @@ def plot_atomic_features(latent_space, method="PCA", dimensions=2, backend="seab
     """
     method = method.lower()
     backend = backend.lower()
-    dot_size = 2.0
+    dot_size = 4.0
 
     supported_methods = ["pca", "tsne"]
 

diff --git a/ml4chem/models/autoencoders.py b/ml4chem/models/autoencoders.py
@@ -464,15 +464,15 @@ def reparameterize(self, mu, logvar):
 
         Parameters
         ----------
-        mu : [type]
-            [description]
-        logvar : [type]
-            [description]
+        mu : tensor
+            Mean values of distribution.
+        logvar : tensor
+            Logarithm of variance of distribution,
 
         Returns
         -------
-        [type]
-            [description]
+        Sample vector
+            A sample from the distribution.
         """
         std = torch.exp(0.5 * logvar)
         eps = torch.randn_like(std)
@@ -490,7 +490,7 @@ def forward(self, X):
 
         Returns
         -------
-        mu and lovar for two multivariate gaussian
+        mu and logvar for two multivariate gaussian
             Decoded latent vector.
         """
 
@@ -875,7 +875,7 @@ def closure(
         device,
         inputs_chunk_vals=None,
         annealing=None,
-        penalize_latent=False
+        penalize_latent=False,
     ):
         """Closure
 
@@ -904,7 +904,7 @@ def closure(
                         device,
                         inputs_chunk_vals,
                         annealing,
-                        penalize_latent
+                        penalize_latent,
                     )
                 )
             )
@@ -933,7 +933,16 @@ def closure(
 
     @classmethod
     def train_batches(
-        Cls, index, chunk, targets, model, lossfxn, device, inputs_chunk_vals, annealing, penalize_latent
+        Cls,
+        index,
+        chunk,
+        targets,
+        model,
+        lossfxn,
+        device,
+        inputs_chunk_vals,
+        annealing,
+        penalize_latent,
     ):
         """A function that allows training per batches
 
@@ -973,12 +982,12 @@ def train_batches(
                 "mus_latent": mus_latent,
                 "logvars_latent": logvars_latent,
                 "annealing": annealing,
+                "input_dimension": model.input_dimension,
             }
         else:
             outputs = model(inputs)
             args = {"outputs": outputs, "targets": targets[index]}
 
-
         # Latent space penalizations
 
         if penalize_latent:
@@ -1001,7 +1010,6 @@ def train_batches(
             # In the case of using EncoderMapLoss the inputs are needed, too.
             args.update({"inputs": inputs_chunk_vals[index]})
 
-
         loss = lossfxn(**args)
         loss.backward()
 

diff --git a/ml4chem/models/loss.py b/ml4chem/models/loss.py
@@ -251,8 +251,15 @@ def get_pairwise_distances(positions, squared=False):
     return distances
 
 
-## def VAELoss(targets, mus_latent, logvars_latent, mus_output, logvars_output):
-def VAELoss(outputs, targets, mus_latent, logvars_latent, annealing, latent=None):
+def VAELoss(
+    outputs,
+    targets,
+    mus_latent,
+    logvars_latent,
+    annealing,
+    latent=None,
+    input_dimension=None,
+):
     """Variational Autoencoder loss function
 
 
@@ -270,6 +277,8 @@ def VAELoss(outputs, targets, mus_latent, logvars_latent, annealing, latent=None
         Contribution of distance loss function to total loss.
     latent : tensor
         The latent space tensor.
+    input_dimension : int
+        Input's dimension.
 
 
     Returns
@@ -278,35 +287,38 @@ def VAELoss(outputs, targets, mus_latent, logvars_latent, annealing, latent=None
         The value of the loss function.
 
     """
-    loss = 0.0
 
+    loss = []
     # LOG_2_PI = np.log(2 * np.pi)
     # loss_rec = LOG_2_PI + torch.sum(logvars_output + (targets - mus_output) ** 2 / (2 * torch.exp(logvars_output)))
 
-    # loss_rec = MSELoss(outputs, targets)
-    # loss_rec = torch.nn.functional.binary_cross_entropy(outputs, targets, reduction='sum')
-
-    loss_rec = 0.5 * torch.nn.functional.mse_loss(outputs, targets, reduction="sum") * 1.
-
-    loss += loss_rec
+    loss_rec = torch.nn.functional.binary_cross_entropy(
+        outputs, targets, reduction="sum"
+    )
+    # criterion = torch.nn.MSELoss(reduction="sum")
+    # loss_rec = criterion(outputs, targets) * input_dimension
+    loss_rec *= input_dimension
+    loss.append(loss_rec)
 
     # see Appendix B from VAE paper:
     # Kingma and Welling. Auto-Encoding Variational Bayes. ICLR, 2014
     # https://arxiv.org/abs/1312.6114
     # 0.5 * sum(1 + log(sigma^2) - mu^2 - sigma^2)
 
+    annealing = 1.0
 
-    annealing = 0.
     kld = (
         -0.5
         * torch.sum(1 + logvars_latent - mus_latent.pow(2) - logvars_latent.exp())
         * annealing
     )
-
-    loss += kld
+    loss.append(kld)
 
     if latent is not None:
         activation_reg = torch.mean(torch.pow(latent, 2))
-        loss += activation_reg
+        loss.append(activation_reg)
+
+    print(loss)
+    loss = torch.mean(torch.stack(loss))
 
     return loss