Skip to content
Permalink
Browse files

update

  • Loading branch information...
astorfi committed May 13, 2019
1 parent fe86a43 commit fa0850d1ba715b28d9ec94715d192fc76ba0fd3b
Showing with 175 additions and 225 deletions.
  1. BIN docs/build/doctrees/content/deep_learning/autoencoder.doctree
  2. BIN docs/build/doctrees/content/deep_learning/mlp.doctree
  3. BIN docs/build/doctrees/content/overview/crossvalidation.doctree
  4. BIN docs/build/doctrees/content/overview/overfitting.doctree
  5. BIN docs/build/doctrees/content/supervised/decisiontrees.doctree
  6. BIN docs/build/doctrees/content/supervised/knn.doctree
  7. BIN docs/build/doctrees/content/supervised/linear_SVM.doctree
  8. BIN docs/build/doctrees/content/supervised/logistic_regression.doctree
  9. BIN docs/build/doctrees/environment.pickle
  10. +33 −47 docs/build/html/_sources/content/deep_learning/autoencoder.rst.txt
  11. +2 −2 docs/build/html/_sources/content/deep_learning/mlp.rst.txt
  12. +12 −6 docs/build/html/_sources/content/overview/crossvalidation.rst.txt
  13. +2 −2 docs/build/html/_sources/content/overview/overfitting.rst.txt
  14. +1 −2 docs/build/html/_sources/content/supervised/decisiontrees.rst.txt
  15. +1 −2 docs/build/html/_sources/content/supervised/knn.rst.txt
  16. +1 −2 docs/build/html/_sources/content/supervised/linear_SVM.rst.txt
  17. +4 −5 docs/build/html/_sources/content/supervised/logistic_regression.rst.txt
  18. +32 −53 docs/build/html/content/deep_learning/autoencoder.html
  19. +4 −4 docs/build/html/content/deep_learning/mlp.html
  20. +33 −19 docs/build/html/content/overview/crossvalidation.html
  21. +5 −9 docs/build/html/content/overview/overfitting.html
  22. +5 −9 docs/build/html/content/supervised/decisiontrees.html
  23. +5 −9 docs/build/html/content/supervised/knn.html
  24. +5 −9 docs/build/html/content/supervised/linear_SVM.html
  25. +12 −16 docs/build/html/content/supervised/logistic_regression.html
  26. +17 −28 docs/build/html/index.html
  27. +1 −1 docs/build/html/searchindex.js
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -6,64 +6,50 @@ Autoencoders
Autoencoders and their implementations in TensorFlow
********************************************************

In this post, you will learn the concept behind Autoencoders as well how
In this post, you will learn the notion behind Autoencoders as well as how
to implement an autoencoder in TensorFlow.

********************************************************
Introduction
********************************************************

Autoencoders are a type of neural networks which copy its input to its
output. They usually consist of two main parts, namely Encoder and
Decoder. The encoder map the input into a hidden layer space which we
refer to as a code. The decoder then reconstructs the input from the
code. There are different types of Autoencoders:

- **Undercomplete Autoencoders:** An autoencoder whose code
dimension is less than the input dimension. Learning such an
autoencoder forces it to capture the most salient features.
However, using a big encoder and decoder in the lack of enough
training data allows the network to memorized the task and omits
learning useful features. In case of having linear decoder, it can
act as PCA. However, adding nonlinear activation functions to the
network makes it a nonlinear generalization of PCA.
- **Regularized Autoencoders:** Rather than limiting the size of
autoencoder and the code dimension for the sake of feature
learning, we can add a loss function to prevent it memorizing the
task and the training data.
- **Sparse Autoencoders:** An autoencoder which has a sparsity
penalty in the training loss in addition to the
reconstruction error. They usually being used for the
porpuse of other tasks such as classification. The loss is
not as straightforward as other regularizers, and we will
discuss it in another post later.
- **Denoising Autoencoders (DAE):** The input of a DAE is a
corrupted copy of the real input which is supposed to be
reconstructed. Therefore, a DAE has to undo the corruption
(noise) as well as reconstruction.
- **Contractive Autoencoders (CAE):** The main idea behind
these type of autoencoders is to learn a representation of
the data which is robust to small changes in the input.
- **Variational Autoencoders:** They maximize the probability of the
training data instead of copying the input to the output and
therefore does not need regularization to capture useful
information.

In this post, we are going to create a simple Undercomplete Autoencoder
in TensorFlow to learn a low dimension representation (code) of the
MNIST dataset.
Autoencoders are a kind of neural networks which imitate their inputs and produce the
exact information at their outputs. They usually include two parts: Encoder and Decoder.
The encoder transforms the input into a hidden space (hidden layer). The decoder then
reconstructs the input information as the output. There are various types of autoencoders:

- **Undercomplete Autoencoders:** In this type, the hidden dimension is smaller than the input dimension.
Training such autoencoder lead to capturing the most prominent features. However, using an overparameterized
architecture in case of a lack of sufficient training data create overfitting and bars learning valuable features.
A linear decoder can operate as PCA. However, the existence of non-linear functions create a more powerful
dimensionality reduction model.
- **Regularized Autoencoders:** Instead of limiting the dimension of an autoencoder and the hidden
layer size for feature learning, a loss function will be added to prevent overfitting.
- **Sparse Autoencoders:** Sparse autoencoders allow for representing the information bottleneck
without demanding a decrease in the size of the hidden layer. Instead, it operates based on a loss
function that penalizes the activations inside a layer.
- **Denoising Autoencoders (DAE):** We want an autoencoder to be sufficiently sensitive to regenerate
the original input but not strictly sensitive so the model can learn a generalizable encoding and decoding.
The approach is to insignificantly corrupt the input data with some noise with an uncorrupted data as the target output..
- **Contractive Autoencoders (CAE):** In this type of autoencoders, for small input variations,
the encoded features should also be very similar. Denoising autoencoders force the reconstruction
function to resist minor changes of the input, while contractive autoencoders enforce the encoder
to resist against the input perturbation.
- **Variational Autoencoders:** A variational autoencoder (VAE) presents a probabilistic fashion
for explaining an observation in hidden space. Therefore, instead of creating an encoder
which results in a value to represent each latent feature, the encoder produces a probability
distribution for each hidden feature.

In this post, we are going to design an Undercomplete Autoencoder
in TensorFlow to train a low dimension representation.

********************************************************
Create an Undercomplete Autoencoder
********************************************************

We are going to create an autoencoder with a 3-layer encoder and 3-layer
decoder. Each layer of encoder downsamples its input along the spatial
dimensions (width, height) by a factor of two using a stride 2.
Consequently, the dimension of the code is 2(width) X 2(height) X
8(depth) = 32 (for an image of 32X32). Similarly, each layer of the
decoder upsamples its input by a factor of two (using transpose
convolution with stride 2).
We are working on building an autoencoder with a 3-layer encoder and 3-layer decoder. Each layer of encoder compresses its input along the spatial
dimensions by a factor of two. Similarly, each segment of the
decoder increases its input dimensionality by a factor of two.

.. code-block:: python
@@ -65,7 +65,7 @@ Thinking of the output in this way won't lose any of the magic of neural network
In MLPs, nodes activate based on **all** of the nodes in the previous layer.


In this example, we let's focus on the single-node layer, which is that way for example purposes.
In this example, let's focus on the single-node layer, which is that way for example purposes.
Each line represents the weights of the nodes in the previous layer.
The sum of the weights of each connection multiplied by the activation of the
connected node results in the activation of our node.
@@ -144,7 +144,7 @@ We discussed how a network learns from training data, specifically using *backpr
We also looked into what defines an MLP network, and how they differ from other neural networks.

*******
Further resources
Further Resources
*******

If you wish to learn more about the topic of neural networks, we recommend this playlist_ by 3Blue1Brown on YouTube.
@@ -1,6 +1,14 @@
Cross-Validation
================

.. contents::
:local:
:depth: 2


Motivation
----------

It's easy to train a model against a particular dataset, but how does
this model perform when introduced with new data? How do you know which
machine learning model to use? Cross-validation answers these questions
@@ -221,14 +229,12 @@ train-test data split is created with the `split()` method:
Note that you can change the P value at the top of the script to see
how different values operate.

.. _holdout.py: https://github.com/machinelearningmindset/machine-learning-course/tree/mastercode/overview/cross-validation/holdout.py
.. _k-fold.py: https://github.com/machinelearningmindset/machine-learning-course/tree/mastercode/overview/cross-validation/k-fold.py
.. _leave-p-out.py: https://github.com/machinelearningmindset/machine-learning-course/tree/mastercode/overview/cross-validation/leave-p-out.py

.. _holdout.py: https://github.com/machinelearningmindset/machine-learning-course/tree/master/code/overview/cross-validation/holdout.py
.. _k-fold.py: https://github.com/machinelearningmindset/machine-learning-course/tree/master/code/overview/cross-validation/k-fold.py
.. _leave-p-out.py: https://github.com/machinelearningmindset/machine-learning-course/tree/master/code/overview/cross-validation/leave-p-out.py

************
References
************
----------

1. https://towardsdatascience.com/cross-validation-in-machine-learning-72924a69872f
2. https://machinelearningmastery.com/k-fold-cross-validation/
@@ -108,9 +108,9 @@ The modules in this section are among the most important in the whole repository
since regardless of the implementation, machine learning always includes these fundamentals.


************
----------
References
************
----------

1. https://machinelearningmastery.com/overfitting-and-underfitting-with-machine-learning-algorithms/
2. https://medium.com/greyatom/what-is-underfitting-and-overfitting-in-machine-learning-and-how-to-deal-with-it-6803a989c76
@@ -318,9 +318,8 @@ whether Mike will go shopping:
prediction_results = tree.predict(encoder.transform(prediction_data))
************
References
************
----------

1. https://towardsdatascience.com/decision-trees-in-machine-learning-641b9c4e8052
2. https://heartbeat.fritz.ai/introduction-to-decision-tree-learning-cd604f85e23
@@ -168,9 +168,8 @@ Eventually the algorithm will classify all the data into 1 class, and there will
.. _Support Vector Machines: https://machine-learning-course.readthedocs.io/en/latest/content/supervised/linear_SVM.html


************
References
************
----------

1. https://medium.com/machine-learning-101/k-nearest-neighbors-classifier-1c1ff404d265
2. https://www.analyticsvidhya.com/blog/2018/03/introduction-k-neighbours-algorithm-clustering/
@@ -174,9 +174,8 @@ the data. You can change it here in the code:
.. _linear_svm.py: https://github.com/machinelearningmindset/machine-learning-course/blob/master/code/supervised/Linear_SVM/linear_svm.py


************
References
************
----------

1. https://www.analyticsvidhya.com/blog/2017/09/understaing-support-vector-machine-example-code/
2. https://stackabuse.com/implementing-svm-and-kernel-svm-with-pythons-scikit-learn/
@@ -29,8 +29,8 @@ Here is the standard logistic function, note that the output is always between
:alt: Logistic
Ref: https://en.wikipedia.org/wiki/Logistic_regression

When to use it
-----------------
When to Use
-----------

Logistic regression is great for situations where you need to classify between two categories.
Some good examples are accepted and rejected applicants and victory or defeat in a competition.
@@ -71,7 +71,7 @@ This formula gives the probability that the input belongs to Class B, which
is the goal of the logistic regression model.


Multinomial logistic regression
Multinomial Logistic Regression
-----------------

Until now, we've been discussing the situation where there are exactly
@@ -144,9 +144,8 @@ As always, keep in mind the basics mentioned in the overview section of this
repository, as there is no fool-proof method for machine learning.


************
References
************
----------

1. https://towardsdatascience.com/logistic-regression-b0af09cdb8ad
2. https://medium.com/datadriveninvestor/machine-learning-model-logistic-regression-5fa4ffde5773
@@ -188,67 +188,46 @@
<h1>Autoencoders<a class="headerlink" href="#autoencoders" title="Permalink to this headline">¶</a></h1>
<div class="section" id="autoencoders-and-their-implementations-in-tensorflow">
<h2>Autoencoders and their implementations in TensorFlow<a class="headerlink" href="#autoencoders-and-their-implementations-in-tensorflow" title="Permalink to this headline">¶</a></h2>
<p>In this post, you will learn the concept behind Autoencoders as well how
<p>In this post, you will learn the notion behind Autoencoders as well as how
to implement an autoencoder in TensorFlow.</p>
</div>
<div class="section" id="introduction">
<h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
<p>Autoencoders are a type of neural networks which copy its input to its
output. They usually consist of two main parts, namely Encoder and
Decoder. The encoder map the input into a hidden layer space which we
refer to as a code. The decoder then reconstructs the input from the
code. There are different types of Autoencoders:</p>
<ul>
<li><p class="first"><strong>Undercomplete Autoencoders:</strong> An autoencoder whose code
dimension is less than the input dimension. Learning such an
autoencoder forces it to capture the most salient features.
However, using a big encoder and decoder in the lack of enough
training data allows the network to memorized the task and omits
learning useful features. In case of having linear decoder, it can
act as PCA. However, adding nonlinear activation functions to the
network makes it a nonlinear generalization of PCA.</p>
</li>
<li><p class="first"><strong>Regularized Autoencoders:</strong> Rather than limiting the size of
autoencoder and the code dimension for the sake of feature
learning, we can add a loss function to prevent it memorizing the
task and the training data.</p>
<blockquote>
<div><ul class="simple">
<li><strong>Sparse Autoencoders:</strong> An autoencoder which has a sparsity
penalty in the training loss in addition to the
reconstruction error. They usually being used for the
porpuse of other tasks such as classification. The loss is
not as straightforward as other regularizers, and we will
discuss it in another post later.</li>
<li><strong>Denoising Autoencoders (DAE):</strong> The input of a DAE is a
corrupted copy of the real input which is supposed to be
reconstructed. Therefore, a DAE has to undo the corruption
(noise) as well as reconstruction.</li>
<li><strong>Contractive Autoencoders (CAE):</strong> The main idea behind
these type of autoencoders is to learn a representation of
the data which is robust to small changes in the input.</li>
</ul>
</div></blockquote>
</li>
<li><p class="first"><strong>Variational Autoencoders:</strong> They maximize the probability of the
training data instead of copying the input to the output and
therefore does not need regularization to capture useful
information.</p>
</li>
<p>Autoencoders are a kind of neural networks which imitate their inputs and produce the
exact information at their outputs. They usually include two parts: Encoder and Decoder.
The encoder transforms the input into a hidden space (hidden layer). The decoder then
reconstructs the input information as the output. There are various types of autoencoders:</p>
<ul class="simple">
<li><strong>Undercomplete Autoencoders:</strong> In this type, the hidden dimension is smaller than the input dimension.
Training such autoencoder lead to capturing the most prominent features. However, using an overparameterized
architecture in case of a lack of sufficient training data create overfitting and bars learning valuable features.
A linear decoder can operate as PCA. However, the existence of non-linear functions create a more powerful
dimensionality reduction model.</li>
<li><strong>Regularized Autoencoders:</strong> Instead of limiting the dimension of an autoencoder and the hidden
layer size for feature learning, a loss function will be added to prevent overfitting.</li>
<li><strong>Sparse Autoencoders:</strong> Sparse autoencoders allow for representing the information bottleneck
without demanding a decrease in the size of the hidden layer. Instead, it operates based on a loss
function that penalizes the activations inside a layer.</li>
<li><strong>Denoising Autoencoders (DAE):</strong> We want an autoencoder to be sufficiently sensitive to regenerate
the original input but not strictly sensitive so the model can learn a generalizable encoding and decoding.
The approach is to insignificantly corrupt the input data with some noise with an uncorrupted data as the target output..</li>
<li><strong>Contractive Autoencoders (CAE):</strong> In this type of autoencoders, for small input variations,
the encoded features should also be very similar. Denoising autoencoders force the reconstruction
function to resist minor changes of the input, while contractive autoencoders enforce the encoder
to resist against the input perturbation.</li>
<li><strong>Variational Autoencoders:</strong> A variational autoencoder (VAE) presents a probabilistic fashion
for explaining an observation in hidden space. Therefore, instead of creating an encoder
which results in a value to represent each latent feature, the encoder produces a probability
distribution for each hidden feature.</li>
</ul>
<p>In this post, we are going to create a simple Undercomplete Autoencoder
in TensorFlow to learn a low dimension representation (code) of the
MNIST dataset.</p>
<p>In this post, we are going to design an Undercomplete Autoencoder
in TensorFlow to train a low dimension representation.</p>
</div>
<div class="section" id="create-an-undercomplete-autoencoder">
<h2>Create an Undercomplete Autoencoder<a class="headerlink" href="#create-an-undercomplete-autoencoder" title="Permalink to this headline">¶</a></h2>
<p>We are going to create an autoencoder with a 3-layer encoder and 3-layer
decoder. Each layer of encoder downsamples its input along the spatial
dimensions (width, height) by a factor of two using a stride 2.
Consequently, the dimension of the code is 2(width) X 2(height) X
8(depth) = 32 (for an image of 32X32). Similarly, each layer of the
decoder upsamples its input by a factor of two (using transpose
convolution with stride 2).</p>
<p>We are working on building an autoencoder with a 3-layer encoder and 3-layer decoder. Each layer of encoder compresses its input along the spatial
dimensions by a factor of two. Similarly, each segment of the
decoder increases its input dimensionality by a factor of two.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">tensorflow.contrib.layers</span> <span class="kn">as</span> <span class="nn">lays</span>

<span class="k">def</span> <span class="nf">autoencoder</span><span class="p">(</span><span class="n">inputs</span><span class="p">):</span>

0 comments on commit fa0850d

Please sign in to comment.
You can’t perform that action at this time.