Merge pull request #3246 from sanuj/cookbook_nn

add cookbook for neural net classification

doc/cookbook/source/examples/neural_nets/feedforward_net_classification.rst

@@ -0,0 +1,55 @@
+======================================
+Feedforward Network for Classification
+======================================
+
+Feedforward network or multi-layer perceptron defines a mapping :math:`y = f(\mathbf{x};\mathbf{\theta})` from an input :math:`\mathbf{x}` to a category :math:`y` and learns the value of parameters :math:`\mathbf{\theta}` by iterative training that results in the best function approximation. The network is a directed acyclic graph composed of an input layer, an output layer and a few hidden layers.
+
+For example,
+
+.. math::
+
+    f(\mathbf{x}) = f^{(3)}(f^{(2)}(f^{(1)}(\mathbf{x})))
+
+where :math:`\mathbf{x}` is the input layer, :math:`f^{(1)}` and :math:`f^{(2)}` are hidden layers and :math:`f^{(3)}` is the output layer.
+
+See chapter 6 in :cite:`Goodfellow-et-al-2016-Book` for a detailed introduction.
+
+-------
+Example
+-------
+
+Imagine we have files with training and test data. We create CDenseFeatures (here 64 bit floats aka RealFeatures) and :sgclass:`CBinaryLabels` as
+
+.. sgexample:: feedforward_net_classification.sg:create_features
+
+We create instances of :sgclass:`CNeuralInputLayer`, :sgclass:`CNeuralLinearLayer` and :sgclass:`NeuralSoftmaxLayer` which are building blocks of :sgclass:`CNeuralNetwork`
+
+.. sgexample:: feedforward_net_classification.sg:add_layers
+
+We create a :sgclass:`CNeuralNetwork` instance by using the above layers and randomly initialize the network parameters by sampling from a gaussian distribution.
+
+.. sgexample:: feedforward_net_classification.sg:create_instance
+
+Before training, we need to set appropriate parameters like regularization coefficient, dropout probabilities, learning rate, etc. as shown below. More parameters can be found in the documentation of :sgclass:`CNeuralNetwork`.
+
+.. sgexample:: feedforward_net_classification.sg:set_parameters
+
+We train the model and apply it to some test data.
+
+.. sgexample:: feedforward_net_classification.sg:train_and_apply
+
+We can extract the parameters of the trained network.
+
+.. sgexample:: feedforward_net_classification.sg:get_params
+
+Finally, we compute accuracy.
+
+.. sgexample:: feedforward_net_classification.sg:evaluate_accuracy
+
+----------
+References
+----------
+:wiki:`Artificial_neural_network`
+
+.. bibliography:: ../../references.bib
+    :filter: docname in docnames

doc/cookbook/source/index.rst

@@ -67,3 +67,12 @@ Distance
    :glob:
 
    examples/distance/**
+
+Neural Networks
+---------------
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+
+   examples/neural_nets/**

doc/cookbook/source/references.bib

@@ -60,7 +60,12 @@ @book{scholkopf2002learning
   year={2002},
   publisher={MIT press}
 }
-
+@unpublished{Goodfellow-et-al-2016-Book,
+    title={Deep Learning},
+    author={I. Goodfellow and Y. Bengio and A. Courville},
+    note={Book in preparation for MIT Press},
+    year={2016}
+}
 @article{escalera2010error,
   title={Error-correcting ouput codes library},
   author={S. Escalera and O. Pujol and P. Radeva},

examples/meta/src/neural_nets/feedforward_net_classification.sg

@@ -0,0 +1,55 @@
+CSVFile f_feats_train("../../data/classifier_binary_2d_nonlinear_features_train.dat")
+CSVFile f_feats_test("../../data/classifier_binary_2d_nonlinear_features_test.dat")
+CSVFile f_labels_train("../../data/classifier_binary_2d_nonlinear_labels_train.dat")
+CSVFile f_labels_test("../../data/classifier_binary_2d_nonlinear_labels_test.dat")
+
+Math:init_random(1)
+
+#![create_features]
+RealFeatures features_train(f_feats_train)
+RealFeatures features_test(f_feats_test)
+BinaryLabels labels_train(f_labels_train)
+BinaryLabels labels_test(f_labels_test)
+#![create_features]
+
+#![add_layers]
+int num_feats = features_train.get_num_features()
+NeuralLayers layers() 
+layers = layers.input(num_feats)
+layers = layers.rectified_linear(10)
+layers = layers.softmax(2)
+DynamicObjectArray all_layers = layers.done()
+#![add_layers]
+
+#![create_instance]
+NeuralNetwork network(all_layers)
+network.quick_connect()
+network.initialize_neural_network()
+#![create_instance]
+
+#![set_parameters]
+network.set_l2_coefficient(0.01)
+network.set_dropout_hidden(0.5)
+network.set_max_num_epochs(50)
+network.set_gd_mini_batch_size(num_feats)
+network.set_gd_learning_rate(0.1)
+network.set_gd_momentum(0.9)
+#![set_parameters]
+
+#![train_and_apply]
+network.set_labels(labels_train)
+network.train(features_train)
+BinaryLabels labels_predict = network.apply_binary(features_test)
+#![train_and_apply]
+
+#![get_params]
+RealVector parameters = network.get_parameters()
+#![get_params]
+
+#![evaluate_accuracy]
+AccuracyMeasure eval()
+real accuracy = eval.evaluate(labels_predict, labels_test)
+#![evaluate_accuracy]
+
+# additional integration testing variables
+RealVector output = labels_predict.get_labels()