New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/neural nets factory and cookbook #4386
Changes from all commits
25506fb
258e25b
9ae858a
42e4c0f
96a3855
5378b9e
db95fb4
11b3922
77aa3ff
a6ddbae
599d431
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
=============================================== | ||
Convolutional Neural Network for Classification | ||
=============================================== | ||
|
||
Convolutional neural network is a class of deep, feed-forward artificial neural networks, most commonly applied to analyzing visual imagery. | ||
The network is a directed acyclic graph composed of an input layer, a few hidden layers can be convolutional layers, pooling layers, fully connected layers or normalization layers and a softmax output layer. | ||
To compute the pre-nonlinearity input to some unit :math:`x_{ij}^{l}` in any layer, we can sum up the contributions from the previous layer cells: | ||
|
||
.. math:: | ||
|
||
x_{ij}^{l}=\sum_{a=0}^{m-1}\sum_{b=0}^{m-1}w_{ab}y_{(i+1)(j+1)}^{l-1} | ||
|
||
where :math:`x_{ij}^{l}` is the input to a neuron in layer :math:`l`, :math:`y_{ij}^{l}` is output of the neuron and :math:`w_{ij}` is it's weight. | ||
See chapter 9 in :cite:`Goodfellow-et-al-2016-Book` for a detailed introduction. | ||
|
||
------- | ||
Example | ||
------- | ||
|
||
Imagine we have files with training and test data. We create CDenseFeatures (here 64 bit floats aka RealFeatures) and :sgclass:`CMulticlassLabels` as | ||
|
||
.. sgexample:: convolutional_net_classification.sg:create_features | ||
|
||
We create a :sgclass:`CNeuralNetwork` instance and randomly initialize the network parameters by sampling from a gaussian distribution. We also set appropriate parameters like regularization coefficient, dropout probabilities, learning rate, etc. as shown below. More parameters can be found in the documentation of :sgclass:`CNeuralNetwork`. | ||
|
||
.. sgexample:: convolutional_net_classification.sg:set_parameters | ||
|
||
.. sgexample:: convolutional_net_classification.sg:create_instance | ||
|
||
We create instances of :sgclass:`CNeuralInputLayer`, :sgclass:`CNeuralConvolutionalLayer` and :sgclass:`NeuralSoftmaxLayer` which are building blocks of :sgclass:`CNeuralNetwork` | ||
|
||
.. sgexample:: convolutional_net_classification.sg:add_layers | ||
|
||
We train the model and apply it to some test data. | ||
|
||
.. sgexample:: convolutional_net_classification.sg:train_and_apply | ||
|
||
Finally, we compute accuracy. | ||
|
||
.. sgexample:: convolutional_net_classification.sg:evaluate_accuracy | ||
|
||
---------- | ||
References | ||
---------- | ||
:wiki:`Convolutional_neural_network` | ||
|
||
.. bibliography:: ../../references.bib | ||
:filter: docname in docnames |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
File f_feats_train = csv_file("../../data/mnist_3class_256d_features_train.dat") | ||
File f_feats_test = csv_file("../../data/mnist_3class_256d_features_test.dat") | ||
File f_labels_train = csv_file("../../data/mnist_3class_256d_labels_train.dat") | ||
File f_labels_test = csv_file("../../data/mnist_3class_256d_labels_test.dat") | ||
|
||
Math:init_random(10) | ||
|
||
#![create_features] | ||
Features features_train = features(f_feats_train) | ||
Features features_test = features(f_feats_test) | ||
Labels labels_train = labels(f_labels_train) | ||
Labels labels_test = labels(f_labels_test) | ||
#![create_features] | ||
|
||
#![create_instance] | ||
Machine network = machine("NeuralNetwork", labels=labels_train, auto_quick_initialize=True, max_num_epochs=4, epsilon=0.0, optimization_method=enum ENNOptimizationMethod.NNOM_GRADIENT_DESCENT, gd_learning_rate=0.01, gd_mini_batch_size=3, max_norm=1.0, dropout_input=0.5) | ||
#![create_instance] | ||
|
||
#![add_layers] | ||
NeuralLayer input = layer("NeuralInputLayer", width=16, height=16, num_neurons=256) | ||
network.add("layers", input) | ||
NeuralLayer conv1 = layer("NeuralConvolutionalLayer", num_maps=3, radius_x=2, radius_y=2, pooling_width=2, pooling_height=2, stride_x=1, stride_y=1) | ||
network.add("layers", conv1) | ||
NeuralLayer conv2 = layer("NeuralConvolutionalLayer", num_maps=3, radius_x=2, radius_y=2, pooling_width=2, pooling_height=2, stride_x=1, stride_y=1) | ||
network.add("layers", conv2) | ||
NeuralLayer softmax = layer("NeuralSoftmaxLayer", num_neurons=3) | ||
network.add("layers", softmax) | ||
#![add_layers] | ||
|
||
#![train_and_apply] | ||
network.train(features_train) | ||
Labels labels_predict = network.apply(features_test) | ||
#![train_and_apply] | ||
|
||
#![evaluate_accuracy] | ||
Evaluation eval = evaluation("MulticlassAccuracy") | ||
real accuracy = eval.evaluate(labels_predict, labels_test) | ||
#![evaluate_accuracy] | ||
|
||
# additional integration testing variables | ||
RealVector output = labels_predict.get_real_vector("labels") |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,55 +1,44 @@ | ||
CSVFile f_feats_train("../../data/classifier_binary_2d_nonlinear_features_train.dat") | ||
CSVFile f_feats_test("../../data/classifier_binary_2d_nonlinear_features_test.dat") | ||
CSVFile f_labels_train("../../data/classifier_binary_2d_nonlinear_labels_train.dat") | ||
CSVFile f_labels_test("../../data/classifier_binary_2d_nonlinear_labels_test.dat") | ||
File f_feats_train = csv_file("../../data/classifier_binary_2d_nonlinear_features_train.dat") | ||
File f_feats_test = csv_file("../../data/classifier_binary_2d_nonlinear_features_test.dat") | ||
File f_labels_train = csv_file("../../data/classifier_binary_2d_nonlinear_labels_train.dat") | ||
File f_labels_test = csv_file("../../data/classifier_binary_2d_nonlinear_labels_test.dat") | ||
|
||
Math:init_random(1) | ||
|
||
#![create_features] | ||
Features features_train = features(f_feats_train) | ||
Features features_test = features(f_feats_test) | ||
BinaryLabels labels_train(f_labels_train) | ||
BinaryLabels labels_test(f_labels_test) | ||
Labels labels_train = labels(f_labels_train) | ||
Labels labels_test = labels(f_labels_test) | ||
#![create_features] | ||
|
||
#![add_layers] | ||
int num_feats = features_train.get_int("num_features") | ||
NeuralLayers layers() | ||
layers.input(num_feats) | ||
layers.rectified_linear(10) | ||
layers.softmax(2) | ||
DynamicObjectArray all_layers = layers.done() | ||
#![add_layers] | ||
|
||
#![create_instance] | ||
NeuralNetwork network(all_layers) | ||
network.quick_connect() | ||
network.initialize_neural_network() | ||
int num_feats = features_train.get_int("num_features") | ||
Machine network = machine("NeuralNetwork", labels=labels_train, auto_quick_initialize=True, l2_coefficient=0.01, dropout_hidden=0.5, max_num_epochs=50, gd_mini_batch_size=num_feats, gd_learning_rate=0.1, gd_momentum=0.9) | ||
#![create_instance] | ||
|
||
#![set_parameters] | ||
network.set_l2_coefficient(0.01) | ||
network.set_dropout_hidden(0.5) | ||
network.set_max_num_epochs(50) | ||
network.set_gd_mini_batch_size(num_feats) | ||
network.set_gd_learning_rate(0.1) | ||
network.set_gd_momentum(0.9) | ||
#![set_parameters] | ||
#![add_layers] | ||
NeuralLayer input = layer("NeuralInputLayer", num_neurons=num_feats) | ||
network.add("layers", input) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yes this is much nicer. Should have thought about it straight away :) |
||
NeuralLayer relu = layer("NeuralRectifiedLinearLayer", num_neurons=10) | ||
network.add("layers", relu) | ||
NeuralLayer softmax = layer("NeuralSoftmaxLayer", num_neurons=2) | ||
network.add("layers", softmax) | ||
#![add_layers] | ||
|
||
#![train_and_apply] | ||
network.set_labels(labels_train) | ||
network.train(features_train) | ||
BinaryLabels labels_predict = network.apply_binary(features_test) | ||
Labels labels_predict = network.apply(features_test) | ||
#![train_and_apply] | ||
|
||
#![get_params] | ||
RealVector parameters = network.get_parameters() | ||
RealVector parameters = network.get_real_vector("params") | ||
#![get_params] | ||
|
||
#![evaluate_accuracy] | ||
AccuracyMeasure am() | ||
real accuracy = am.evaluate(labels_predict, labels_test) | ||
Evaluation eval = evaluation("AccuracyMeasure") | ||
real accuracy = eval.evaluate(labels_predict, labels_test) | ||
#![evaluate_accuracy] | ||
|
||
# additional integration testing variables | ||
RealVector output = labels_predict.get_labels() | ||
RealVector output = labels_predict.get_real_vector("labels") |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,54 +1,44 @@ | ||
CSVFile f_feats_train("../../data/regression_1d_sinc_features_train_with_9d_noise.dat") | ||
CSVFile f_feats_test("../../data/regression_1d_sinc_features_test_with_9d_noise.dat") | ||
CSVFile f_labels_train("../../data/regression_1d_sinc_labels_train.dat") | ||
CSVFile f_labels_test("../../data/regression_1d_sinc_labels_test.dat") | ||
File f_feats_train = csv_file("../../data/regression_1d_sinc_features_train_with_9d_noise.dat") | ||
File f_feats_test = csv_file("../../data/regression_1d_sinc_features_test_with_9d_noise.dat") | ||
File f_labels_train = csv_file("../../data/regression_1d_sinc_labels_train.dat") | ||
File f_labels_test = csv_file("../../data/regression_1d_sinc_labels_test.dat") | ||
|
||
Math:init_random(1) | ||
|
||
#![create_features] | ||
Features features_train = features(f_feats_train) | ||
Features features_test = features(f_feats_test) | ||
RegressionLabels labels_train(f_labels_train) | ||
RegressionLabels labels_test(f_labels_test) | ||
Labels labels_train = labels(f_labels_train) | ||
Labels labels_test = labels(f_labels_test) | ||
#![create_features] | ||
|
||
#![add_layers] | ||
int dimensions = features_train.get_int("num_features") | ||
NeuralLayers layers() | ||
layers.input(dimensions) | ||
layers.rectified_linear(20) | ||
layers.linear(1) | ||
DynamicObjectArray all_layers = layers.done() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this taken care of? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ah yes it just returns the DynamicObjectArray so if we set that instead using add, all should be good |
||
#![add_layers] | ||
|
||
#![create_instance] | ||
NeuralNetwork network(all_layers) | ||
network.quick_connect() | ||
network.initialize_neural_network() | ||
int num_feats = features_train.get_int("num_features") | ||
Machine network = machine("NeuralNetwork", labels=labels_train, auto_quick_initialize=True, l2_coefficient=0.1, epsilon=0.0, max_num_epochs=40, gd_learning_rate=0.1, gd_momentum=0.9) | ||
#![create_instance] | ||
|
||
#![set_parameters] | ||
network.set_l2_coefficient(0.1) | ||
network.set_max_num_epochs(40) | ||
network.set_epsilon(0.0) | ||
network.set_gd_learning_rate(0.1) | ||
network.set_gd_momentum(0.9) | ||
#![set_parameters] | ||
#![add_layers] | ||
NeuralLayer input = layer("NeuralInputLayer", num_neurons=num_feats) | ||
network.add("layers", input) | ||
NeuralLayer relu = layer("NeuralRectifiedLinearLayer", num_neurons=20) | ||
network.add("layers", relu) | ||
NeuralLayer linear = layer("NeuralLinearLayer", num_neurons=1) | ||
network.add("layers", linear) | ||
#![add_layers] | ||
|
||
#![train_and_apply] | ||
network.set_labels(labels_train) | ||
network.train(features_train) | ||
RegressionLabels labels_predict = network.apply_regression(features_test) | ||
Labels labels_predict = network.apply(features_test) | ||
#![train_and_apply] | ||
|
||
#![get_params] | ||
RealVector parameters = network.get_parameters() | ||
RealVector parameters = network.get_real_vector("params") | ||
#![get_params] | ||
|
||
#![evaluate_error] | ||
MeanSquaredError err() | ||
real mse = err.evaluate(labels_predict, labels_test) | ||
#![evaluate_error] | ||
#![evaluate_accuracy] | ||
Evaluation eval = evaluation("MeanSquaredError") | ||
real mse = eval.evaluate(labels_predict, labels_test) | ||
#![evaluate_accuracy] | ||
|
||
# additional integration testing variables | ||
RealVector output = labels_predict.get_labels() | ||
RealVector output = labels_predict.get_real_vector("labels") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if you change these references you will need to change the cookbook ones as well! It refers to these snippets!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
right i will edit the cookbook too :)