Adding description of softmac activation in Logistic Regreesion Page (#…

…86)

docs/logistic_regression.rst

@@ -313,7 +313,16 @@ For each sub-problem, we select one class (YES) and lump all the others into a s
 Softmax activation
 ------------------
 
-something about softmax here...
+The softmax function (softargmax or normalized exponential function) is a function that takes as input a vector of K real numbers, and normalizes it into a probability distribution consisting of K probabilities proportional to the exponentials of the input numbers. That is, prior to applying softmax, some vector components could be negative, or greater than one; and might not sum to 1; but after applying softmax, each component will be in the interval [ 0 , 1 ] , and the components will add up to 1, so that they can be interpreted as probabilities.
+The standard (unit) softmax function is defined by the formula 
+
+.. math::
+
+  \begin{align}
+   σ(z_i) = \frac{e^{z_{(i)}}}{\sum_{j=1}^K e^{z_{(j)}}}\ \ \ for\ i=1,.,.,.,K\ and\ z=z_1,.,.,.,z_K
+  \end{align}
+
+In words: we apply the standard exponential function to each element :math:`z_i` of the input vector :math:`z` and normalize these values by dividing by the sum of all these exponentials; this normalization ensures that the sum of the components of the output vector :math:`σ(z)` is 1. [9]_
 
 
 Scikit-Learn example
@@ -337,3 +346,4 @@ Let's compare our performance to the ``LogisticRegression`` model provided by sc
 .. [6] http://math.stackexchange.com/questions/78575/derivative-of-sigmoid-function-sigma-x-frac11e-x
 .. [7] https://en.wikipedia.org/wiki/Monotoniconotonic_function
 .. [8] http://scikit-learn.org/stable/modules/linear_model.html#logistic-regression>
+.. [9] https://en.wikipedia.org/wiki/Softmax_function