Multi-Layer Feed-Forward Neural Network
FFNN class contains a fully-connected, 3-layer feed-forward neural network. This neural net uses a standard backpropagation training algorithm (stochastic gradient descent), and is designed for flexibility and use in performance-critical applications.
FFNN instance is easy...
let network = FFNN(inputs: 100, hidden: 64, outputs: 10, learningRate: 0.7, momentum: 0.4, weights: nil, activationFunction : .Sigmoid, errorFunction: .CrossEntropy(average: false))
You must provide eight parameters to the initializer:
inputs: The number of input nodes (aka 'neurons'). This number corresponds to the dimensionality of the data that you plan to feed the network. If the above example were to be used for handwriting recognition,
100might be the number of pixels in each image being processed.
hidden: The number of nodes in the hidden layer. The ideal number of hidden nodes depends heavily on the application, and should be determined by testing. If you're completely unsure, [(inputs * 2/3) + outputs] might be a good place to start.
outputs: The number of output nodes. For classification problems (like recognizing handwritten digits), the number of outputs usually corresponds to the number of possible classifications. In this example, each output might correspond to one digit (0-9). The number of outputs depends entirely on the problem being applied.
learningRate: The 'learning rate' to apply during the backpropagation phase of training. If you're unsure what this means,
0.7is probably a good number.
momentum: Another constant applied during backpropagation. If you're not sure, try
weights: An optional array of
Floats used to initialize the weights of the neural network. This allows you to 'clone' a pre-trained network, and begin solving problems without training first. When you're creating a new network from scratch, leave this parameter
niland random weights will calculated based on your input data.
activationFunction: One of the supported
ActivationFunctions to apply to the hidden and output nodes.
errorFunction: One of the supported
ErrorFunctions for calculating error on a validation set during training.
Alternatively, the following methods may be used to read/write a neural network from disk:
fromFile- A static method used to initialize a
FFNNfrom file. This is the easiest way to package an application with a pre-trained neural network.
This method accepts either an
NSURL with the full filepath, or a
String specifying only the filename. When a filename is given, it is assumed that the file resides in the user's default documents directory. This file will usually have been generated using the
writeToFile() method below.
let network = FFNN.fromFile(fileURL)
writeToFile- Writes the current state of the
FFNNto the specified file. This includes the structure of the neural network itself, all of its current weights (preserving any training that has been performed), and all parameters (such as learning rate, activation function, etc.) that have been set.
fromFile(), this method accepts either an
NSURL for a custom filepath, or a
String with the desired filename to reside in the user's default documents directory.
You update your
FFNN using this method:
update- Accepts a single set of input data, and returns the resulting output as calculated by the neural net.
let output: [Float] = try network.update(inputs: imagePixels)
FFNN using the following methods:
backpropagate- Used to train the network manually. Accepts the single set of expected outputs (aka 'answers') corresponding to the most recent
updatecall. Returns the total error, as calculated from the difference between the expected and actual outputs.
let error: Float = try network.backpropagate(answer: correctAnswer)
train- Initiates an automated training process on the neural network. Accepts all sets of inputs and corresponding answers to use during the training process and all sets of inputs and answers to be used for network validation, as well as an error threshold to determine when a sufficient solution has been found.
The validation data (
testAnswers) will NOT be used to train the network, but will be used to test the network's progress periodically. Once the desired error threshold on the validation data has been reached, the training will stop. Ideally, the validation data should be randomly selected and representative of the entire search space.
Note: this method will block the calling thread until it is finished, but may safely be dispatched to a background queue.
let weights = try network.train(inputs: allImages, answers: allAnswers, testInputs: validationImages, testAnswers: validationAnswers, errorThreshold: 0.2)
A few methods and properties are provided to modify the state of the neural network:
getWeights- Returns a serialized array of the network's current weights.
let weights = network.getWeights()
resetWithWeights- Allows the user to reset the network with specified weights. Accepts a serialized array of weights, as returned by the
momentumFactor are both mutable properties on
FFNN that may be safely tuned at any time.
To achieve nonlinearity,
FFNN uses a one of several activation functions for hidden and output nodes, as configured during initialization. Because of this property, you will achieve better results if the following points are taken into consideration:
- Input data should generally be normalized to have a mean of
0and standard deviation of
- Except in the case of Linear activation, outputs will always reside in the range (0, 1). For regression problems, a wider range is often needed and thus the outputs must be scaled accordingly.
- When providing 'answers' for backpropagation, this data must be scaled in reverse so that all outputs also reside in the range (0, 1). Again, this does not apply to networks using linear activation functions.
Softmax activation will be added soon.