reformatted activation function section

code/activation_functions.py

@@ -2,15 +2,13 @@
 import numpy as np
 
 def relu(z):
-    return max(0, z)
+  return max(0, z)
 
 def relu_prime(z):
-    if z > 0:
-        return 1
-    return 0
+  return 1 if z > 0 else 0
 
 def sigmoid(z):
-    return 1.0 / (1 + np.exp(-z))
+  return 1.0 / (1 + np.exp(-z))
 
 def sigmoid_prime(z):
-    return sigmoid(z) * (1 - sigmoid(z))
+  return sigmoid(z) * (1-sigmoid(z))

docs/activation_functions.rst

@@ -7,6 +7,7 @@ Activation Functions
 .. contents:: :local:
 
 
+
 ELU
 ===
 
@@ -23,65 +24,74 @@ Be the first to `contribute! <https://github.com/bfortuner/ml-cheatsheet>`__
 ReLU
 ====
 
-.. image:: images/relu.png
-    :align: center
-
 A recent invention which stands for Rectified Linear Units. The formula is deceptively simple: :math:`max(0,z)`. Despite its name and appearance, it’s not linear and provides the same benefits as Sigmoid but with better performance.
 
-.. math::
++-------------------------------------------------------+------------------------------------------------------+
+| Function                                              | Derivative                                           |
++-------------------------------------------------------+------------------------------------------------------+
+| .. math::                                             | .. math::                                            |
+|      R(z) = \begin{Bmatrix} z & z > 0 \\              |       R'(z) = \begin{Bmatrix} 1 & z>0 \\             |
+|       0 & z <= 0 \end{Bmatrix}                        |       0 & z<0 \end{Bmatrix}                          |
++-------------------------------------------------------+------------------------------------------------------+
+| .. image:: images/relu.png                            | .. image:: images/relu_prime.png                     |
+|       :align: center                                  |      :align: center                                  |
+|       :width: 256 px                                  |      :width: 256 px                                  |
+|       :height: 256 px                                 |      :height: 256 px                                 |
++-------------------------------------------------------+------------------------------------------------------+
+| .. literalinclude:: ../code/activation_functions.py   | .. literalinclude:: ../code/activation_functions.py  |
+|       :pyobject: relu                                 |      :pyobject: relu_prime                           |
++-------------------------------------------------------+------------------------------------------------------+
 
-  R(z) = \begin{Bmatrix}
-  z & z > 0 \\
-  0 & otherwise \\
-  \end{Bmatrix}\\
+.. quick create tables with tablesgenerator.com/text_tables and import our premade template in figures/
 
-.. literalinclude:: ../code/activation_functions.py
-    :language: python
-    :pyobject: relu
+.. rubric:: Pros
 
-**Derivative**
+- Pro 1
 
-The derivative of relu...
+.. rubric:: Cons
 
-.. math::
+- Con 1
 
-  R'(z) = \begin{Bmatrix}
-  1 & z>0 \\
-  0 & z<0 \\
-  \end{Bmatrix}
+.. rubric:: Further reading
 
-.. literalinclude:: ../code/activation_functions.py
-    :language: python
-    :pyobject: relu_prime
+- `Deep Sparse Rectifier Neural Networks <http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf>`_ Glorot et al., (2011)
+- `Yes You Should Understand Backprop <https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b>`_, Karpathy (2016)
 
 
 .. _activation_sigmoid:
 
 Sigmoid
 =======
 
-.. image:: images/sigmoid.png
-    :align: center
-
 Sigmoid takes a real value as input and outputs another value between 0 and 1. It’s easy to work with and has all the nice properties of activation functions: it’s non-linear, continuously differentiable, monotonic, and has a fixed output range.
 
-.. math::
++-----------------------------------------------------+-----------------------------------------------------+
+| Function                                            | Derivative                                          |
++-----------------------------------------------------+-----------------------------------------------------+
+| .. math::                                           | .. math::                                           |
+|      S(z) = \frac{1} {1 + e^{-z}}                   |      S'(z) = S(z) \cdot (1 - S(z))                  |
++-----------------------------------------------------+-----------------------------------------------------+
+| .. image:: images/sigmoid.png                       | .. image:: images/sigmoid_prime.png                 |
+|       :align: center                                |       :align: center                                |
+|       :width: 256 px                                |       :width: 256 px                                |
++-----------------------------------------------------+-----------------------------------------------------+
+| .. literalinclude:: ../code/activation_functions.py | .. literalinclude:: ../code/activation_functions.py |
+|       :pyobject: sigmoid                            |       :pyobject: sigmoid_prime                      |
++-----------------------------------------------------+-----------------------------------------------------+
+
+.. quick create tables with tablesgenerator.com/text_tables and import our premade template in figures/
 
-  S(z) = \frac{1} {1 + e^{-z}}
+.. rubric:: Pros
 
-.. literalinclude:: ../code/activation_functions.py
-    :language: python
-    :pyobject: sigmoid
+- Pro 1
 
-**Derivative**
+.. rubric:: Cons
 
-.. math::
+- Con 1
 
-  S'(z) = S(z) * (1 - S(z))
+.. rubric:: Further reading
 
-.. literalinclude:: ../code/activation_functions.py
-    :language: python
-    :pyobject: sigmoid_prime
+- `Yes You Should Understand Backprop <https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b>`_, Karpathy (2016)
 
 
 Softmax

docs/figures/activation_function_table.tgn

@@ -0,0 +1 @@
+{"rows_views":[[{"style":{"borders":"","font_style":{},"text_color":"","bg_color":"","halign":"left","valign":"middle","padding":{"top":10,"right":5,"bottom":10,"left":5}}},{"style":{"borders":"","font_style":{},"text_color":"","bg_color":"","halign":"left","valign":"middle","padding":{"top":10,"right":5,"bottom":10,"left":5}}}],[{"style":{"borders":"","font_style":{},"text_color":"","bg_color":"","halign":"left","valign":"middle","padding":{"top":10,"right":5,"bottom":10,"left":5}}},{"style":{"borders":"","font_style":{},"text_color":"","bg_color":"","halign":"left","valign":"middle","padding":{"top":10,"right":5,"bottom":10,"left":5}}}],[{"style":{"borders":"lrtb","font_style":{},"text_color":"","bg_color":"","halign":"left","valign":"top","padding":{"top":10,"right":5,"bottom":10,"left":5}}},{"style":{"borders":"lrtb","font_style":{},"text_color":"","bg_color":"","halign":"left","valign":"top","padding":{"top":10,"right":5,"bottom":10,"left":5}}}],[{"style":{"borders":"lrtb","font_style":{},"text_color":"","bg_color":"","halign":"left","valign":"top","padding":{"top":10,"right":5,"bottom":10,"left":5}}},{"style":{"borders":"lrtb","font_style":{},"text_color":"","bg_color":"","halign":"left","valign":"top","padding":{"top":10,"right":5,"bottom":10,"left":5}}}]],"model":{"rows":[[{"value":"Function","cspan":1,"rspan":1},{"value":"Derivative","cspan":1,"rspan":1}],[{"value":".. math::\n      r(x) = x + 1","cspan":1,"rspan":1},{"value":".. math::\n       r(x) = x + 1","cspan":1,"rspan":1}],[{"value":".. image:: images/sigmoid.png\n      :align: center\n      :width: 256 px\n      :height: 256 px","cspan":1,"rspan":1},{"value":".. image:: images/sigmoid_prime.png\n      :align: center\n      :width: 256 px\n      :height: 256 px","cspan":1,"rspan":1}],[{"value":".. literalinclude:: ../code/activation_functions.py\n      :pyobject: sigmoid","cspan":1,"rspan":1},{"value":".. literalinclude:: ../code/activation_functions.py\n      :pyobject: sigmoid_prime\n","cspan":1,"rspan":1}]]},"theme":null,"fixed_layout":false}

docs/logistic_regression.rst

@@ -9,7 +9,7 @@ Logistic Regression
 Introduction
 ============
 
-Logistic regression is a classification algorithm used to assign observations to a discrete set of classes. Unlike linear regression which outputs number values on a continuous plane, logistic regression transforms its output using the logistic sigmoid function to return a probability value which can then be mapped to two or more discrete classes.
+Logistic regression is a classification algorithm used to assign observations to a discrete set of classes. Unlike linear regression which outputs continuous number values, logistic regression transforms its output using the logistic sigmoid function to return a probability value which can then be mapped to two or more discrete classes.
 
 Comparison to linear regression
 -------------------------------