Skip to content

Commit 64e8e7e

Browse files
amroamroamroamroamroamro
committed
Merge pull request #3 from StackOverflowMATLABchat/add-docstring
Added in comprehensive docstring and added in minor changes
2 parents cde7829 + f4e5b59 commit 64e8e7e

File tree

2 files changed

+215
-34
lines changed

2 files changed

+215
-34
lines changed

‎NeuralNet2.m‎

Lines changed: 214 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,118 @@
11
classdef NeuralNet2 < handle
2+
% NeuralNet2 Neural Network implementation for the NeuralNetPlayground tool
3+
% This class implements the training (forward and backpropagation) and
4+
% predictions using Artificial Neural Networks (ANN). The primary purpose
5+
% is to assist in the construction of the NeuralNetPlayground framework
6+
% as well as providing a framework for training neural networks that
7+
% uses core MATLAB functionality only (i.e. no toolbox dependencies).
8+
%
9+
% This class is initialized by specifying the total number of input
10+
% layer neurons, hidden layer neurons for each hidden layer desired
11+
% and the total number of output layer neurons as a single vector.
12+
% With specified training examples and expected outputs, the neural
13+
% network weights are learned with Stochastic Gradient Descent.
14+
% The learned weights can be used to predict new examples given the
15+
% learned weights.
16+
%
17+
% NeuralNet2 Properties:
18+
% LearningRate - The learning rate for Stochastic Gradient Descent
19+
% Must be strictly positive (i.e. > 0).
20+
% Default value is 0.03.
21+
% ActivationFunction - The activation function to be applied to each
22+
% neuron in the hidden and output layer.
23+
% The ones you can choose from are:
24+
% 'linear': Linear
25+
% 'relu': Rectified Linear Unit (i.e. ramp)
26+
% 'tanh': Hyperbolic Tangent
27+
% 'sigmoid': Sigmoidal
28+
% Default is 'tanh'.
29+
%
30+
% RegularizationType - Apply regularization to the training process
31+
% if desired.
32+
% The ones you can choose from are:
33+
% 'L1': L1 regularization
34+
% 'L2': L2 regularization
35+
% 'none': No regularization
36+
% Default is 'none'
37+
%
38+
% RegularizationRate - The rate of regularization to apply (if desired)
39+
% Must be non-negative (i.e. >= 0).
40+
% Default is 0.
41+
%
42+
% BatchSize - Number of training examples selected per epoch
43+
% Choosing 1 example would implement true Stochastic
44+
% Gradient Descent while choosing the total number
45+
% of examples would implement Batch Gradient Descent.
46+
% Choosing any value in between implements mini-batch
47+
% Stochastic Gradient Descent.
48+
% Must be strictly positive (i.e. > 0) and integer.
49+
% Default is 10.
50+
%
51+
% Example Use
52+
% -----------
53+
% X = [0 0; 0 1; 1 0; 1 1]; % Define XOR data
54+
% Y = [-1; 1; 1; -1];
55+
% net = NeuralNet2([2 2 1]); % Create Neural Network object
56+
% N = 5000; % Do 5000 iterations of Stochastic Gradient Descent
57+
%
58+
% % Customize Neural Network engine
59+
% net.LearningRate = 0.1; % Learning rate is set to 0.1
60+
% net.RegularizationType = 'L2'; % Regularization is L2
61+
% net.RegularizationRate = 0.001; % Regularization rate is 0.01
62+
%
63+
% perf = net.train(X, Y, N); % Train the Neural Network
64+
% Ypred = net.sim(X); % Use trained object on original examples
65+
% plot(1:N, perf); % Plot cost function per epoch
66+
%
67+
% % Display results
68+
% disp('Training Examples and expected labels'); display(X); display(Y);
69+
% disp('Predicted outputs'); display(Ypred);
70+
%
71+
% See also NEURALNETAPP
72+
%
73+
% StackOverflowMATLABchat - http://chat.stackoverflow.com/rooms/81987/matlab-and-octave
74+
% Authors: Raymond Phan - http://stackoverflow.com/users/3250829/rayryeng
75+
% Amro - http://stackoverflow.com/users/97160/amro
276

377
properties (Access = public)
4-
LearningRate
5-
ActivationFunction
6-
RegularizationType
7-
RegularizationRate
8-
BatchSize
78+
LearningRate % The learning rate (positive number)
79+
ActivationFunction % The desired activation function (string)
80+
RegularizationType % The type of regularization (string)
81+
RegularizationRate % The regularization rate (non-negative number)
82+
BatchSize % The size of the batch per epoch (positive integer number)
983
end
1084

1185
properties (Access = private)
12-
inputSize
13-
hiddenSizes
14-
outputSize
15-
weights
86+
inputSize % Single value denoting how many neurons are in the input layer
87+
hiddenSizes % Vector denoting how many neurons per hidden layer
88+
outputSize % Single value denoting how many neurons are in the output layer
89+
weights % The weights of the neural network per layer
1690
end
1791

1892
methods
19-
% Class constructor
2093
function this = NeuralNet2(layerSizes)
94+
% NeuralNet2 Create a Neural Network Instance
95+
% The constructor takes in a vector of layer sizes where the
96+
% first element denotes how many neurons are in the input layer,
97+
% the next N elements denote how many neurons are in each desired
98+
% hidden layer and the last element denotes how many neurons are
99+
% in the output layer. Take note that the amount of neurons
100+
% per layer that you specify does not include the bias units.
101+
% These will be included when training the network. Therefore,
102+
% the expected size of the vector is N + 2 where N is the total
103+
% number of hidden layers for the neural network.
104+
%
105+
% The following example creates a neural network with 1 input
106+
% neuron (plus a bias) in the input layer, 2 hidden neurons
107+
% (plus a bias) in the first hidden layer, 3 hidden neurons
108+
% (plus a bias) in the second hidden layer and 1 output neuron
109+
% in the output layer
110+
%
111+
% layerSizes = [1 2 3 1];
112+
% net = NeuralNet2(layerSizes);
113+
21114
% default params
22-
this.LearningRate = 0.003;
115+
this.LearningRate = 0.03;
23116
this.ActivationFunction = 'Tanh';
24117
this.RegularizationType = 'None';
25118
this.RegularizationRate = 0;
@@ -62,30 +155,81 @@ function configure(this, X, Y)
62155
outMax = max(Y);
63156
end
64157

65-
% Initialize neural network weights
66158
function init(this)
67-
% initialize with random weights
159+
% init Initialize the Neural Network Weights
160+
% This method initializes the neural network weights
161+
% for connections between neurons so that all weights
162+
% are within the range of [-0.5,0.5]
163+
%
164+
% Uses:
165+
% net = NeuralNet2([1 2 1]);
166+
% net.init();
167+
68168
for i=1:numel(this.weights)
69169
num = numel(this.weights{i});
70170
this.weights{i}(:) = rand(num,1) - 0.5; % [-0.5,0.5]
71171
end
72172
end
73173

74-
% Perform training with Stochastic Gradient Descent
75174
function perf = train(this, X, Y, numIter)
76-
175+
% train Perform neural network training with Stochastic Gradient Descent (SGD)
176+
% This method performs training on the neural network structure
177+
% that was specified when creating an instance of the class.
178+
% Using training example features and their expected outcomes,
179+
% trained network weights are created to facilitate future
180+
% predictions.
181+
%
182+
% Inputs:
183+
% X - Training example features as a 2D matrix of size M x N
184+
% M is the total number of examples and N are the
185+
% total number of features. M is expected to be the
186+
% same size as the number of input layer neurons.
187+
%
188+
% Y - Training example expected outputs as a 2D matrix of size
189+
% M x P where M is the total number of examples and P is
190+
% the total number of output neurons in the output layer.
191+
%
192+
% numIter - Number of iterations Stochastic Gradient Descent
193+
% should take while training. This is an optional
194+
% parameter and the number of iterations defaults to
195+
% 1 if omitted.
196+
%
197+
% Outputs:
198+
% perf - An array of size numIter x 1 which denotes
199+
% the cost between the predicted outputs and
200+
% expected outputs at each iteration of learning
201+
% the weights
202+
%
203+
% Uses:
204+
% net = NeuralNet2([1 2 1]); % Create NN object
205+
% % Create example data here stored in X and Y
206+
% %...
207+
% %...
208+
% perf = net.train(X, Y); % Perform 1 iteration
209+
% perf2 = net.train(X, Y, 500); % Perform 500 iterations
210+
%
211+
% See also NEURALNET2, SIM
212+
213+
% If the number of iterations is not specified, assume 1
77214
if nargin < 4, numIter = 1; end
78215

79216
% Ensure correct sizes
80-
assert(size(X,1) == size(Y,1))
217+
assert(size(X,1) == size(Y,1), ['Total number of examples ' ...
218+
'the inputs and outputs should match'])
81219

82220
% Ensure regularization rate and batch size is proper
83-
assert(this.BatchSize >= 1);
84-
assert(this.RegularizationRate >= 0);
221+
assert(this.BatchSize >= 1, 'Batch size should be 1 or more');
222+
assert(this.RegularizationRate >= 0, ['Regularization rate ' ...
223+
'should be 0 or larger']);
85224

86225
% Check if we have specified the right regularization type
87226
regType = this.RegularizationType;
88-
assert(any(strcmpi(regType, {'l1', 'l2', 'none'})));
227+
assert(any(strcmpi(regType, {'l1', 'l2', 'none'})), ...
228+
['Ensure that you choose one of ''l1'', ''l2'' or '...
229+
'''none'' for the regularization type']);
230+
231+
% Ensure number of iterations is strictly positive
232+
assert(numIter >= 1, 'Number of iterations should be positive');
89233

90234
% Initialize cost function array
91235
perf = zeros(1, numIter);
@@ -97,7 +241,8 @@ function init(this)
97241
L = numel(this.weights);
98242

99243
% Get batch size
100-
B = this.BatchSize;
244+
% Remove decimal places in case of improper input
245+
B = floor(this.BatchSize);
101246

102247
% Safely catch if batch size is larger than total number
103248
% of examples
@@ -132,8 +277,6 @@ function init(this)
132277
% Get activation function
133278
fcn = getActivationFunction(this.ActivationFunction);
134279

135-
skipFactor = floor(numIter/10);
136-
137280
% For each iteration...
138281
for ii = 1:numIter
139282
% If the batch size is equal to the total number of examples
@@ -225,7 +368,7 @@ function init(this)
225368
indwp = w > 0;
226369
indwn = w < 0;
227370

228-
% 2c - Perform the udpate on each condition
371+
% 2c - Perform the update on each condition
229372
% individually
230373
w(indwp) = max(0, w(indwp) - (uk + q(indwp)));
231374
w(indwn) = min(0, w(indwn) + (uk - q(indwn)));
@@ -256,23 +399,64 @@ function init(this)
256399
end
257400
end
258401

259-
% Perform forward propagation
260-
% Note that the bias units are the last row of the matrix
261-
% Inputs are in a 2D matrix of N x M
262-
% N is the number of examples
263-
% M is the number of features / number of input neurons
264402
function [OUT,OUTS] = sim(this, X)
403+
% sim Perform Neural Network Predictions
404+
% This method performs forward propagation using the
405+
% learned weights after training. Forward propagation
406+
% uses the learned weights to propogate information
407+
% throughout the neural network and the predicted outcome
408+
% is seen at the output layer.
409+
%
410+
% Inputs:
411+
% X - Training examples to predict their outcomes
412+
% This is a M x N matrix where M is the total number of
413+
% examples and N is the total number of features.
414+
% N must equal to the total number of neurons in the input
415+
% layer
416+
%
417+
% Outputs:
418+
% OUT - The predicted outputs using the training examples in X.
419+
% This is a M x P matrix where M is the total number of
420+
% training examples and P is the total number of output
421+
% neurons
422+
%
423+
% OUTS - This is a 1D cell array of size 1 x NN where NN
424+
% is the total number of layers in the neural network
425+
% including the input and output layers. Therefore, if
426+
% K is equal to the total number of hidden layers,
427+
% NN = K+2. This cell array contains the outputs of
428+
% each example per layer. Specifically, each element
429+
% OUTS{ii} would be a M x Q matrix where Q would be the
430+
% total number of neurons in layer ii without the bias
431+
% unit. ii=1 is the hidden layer and ii=NN is the output
432+
% layer. OUTS{ii} would contain all of the outputs for
433+
% each example in layer ii.
434+
%
435+
% Uses:
436+
% net = NeuralNet2([1 2 1]); % Create NN object
437+
% % Create your training data and train your neural network
438+
% % here...
439+
% % Also create test data here stored in XX...
440+
% % ...
441+
% OUT = net.sim(XX); % Find predictions
442+
% [OUT,OUTS] = net.sim(XX); % Find predictions and outputs
443+
% % per layer
444+
%
445+
% See also NEURALNET2, TRAIN
446+
265447
% Check if the total number of features matches the
266448
% total number of input neurons
267-
assert(size(X,2) == this.inputSize);
449+
assert(size(X,2) == this.inputSize, ['Number of features '...
450+
'should match the number of input neurons']);
268451

269452
% Get total number of examples
270453
N = size(X,1);
271454

272455
%%% Begin algorithm
273-
% Start with first layer
456+
% Start with input layer
274457
OUT = X;
275458

459+
% Also initialize cell array with input layer's contents
276460
OUTS = cell(1,numel(this.weights)+1);
277461
OUTS{1} = OUT;
278462

@@ -288,7 +472,6 @@ function init(this)
288472
end
289473
end
290474
end
291-
292475
end
293476

294477
function fcn = getActivationFunction(activation)

‎demos/demo_nn.m‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
%% Neural Network Demo
22
% Demonstrates how to use |NeuralNet| class for binary classification.
3-
%
43

54
%% Data
65
% Create XOR dataset
@@ -33,9 +32,8 @@
3332
net.BatchSize = 10;
3433
display(net)
3534

36-
%% Find optimal weights
35+
%% Training and Testing Network
3736

38-
% train network
3937
N = 5000; % number of iterations
4038
disp('Training...'); tic
4139
costVal = net.train(Xtrain, Ytrain, N);

0 commit comments

Comments
 (0)