Using the Network (Repo Code)

This is not how you should use the network if you included the NuGet Package. Please use the guide here. This guide is for people who pulled down the repository to get all the newer functionality that is not yet in NuGet (or not fully tested).

Creating the network

The constructor takes no arguments. You will build the network layer by layer before training.

Once you create a new object with something like

FullyConnectedNetwork _network = new FullyConnectedNetwork();

You simply call the AddLayer() method to create a new layer in the network. The weights and biases will be automatically initialized for you as you create the network. To add a layer, you simply say how many nodes will be in that layer, and what activation function you want to be used for that layer. Here are a few examples showing the 4 different activation functions:

_network.AddLayer(784, NonLinearFunction.Sigmoid);
_network.AddLayer(500, NonLinearFunction.Tanh);
_network.AddLayer(100, NonLinearFunction.ReLU);
_network.AddLayer(10, NonLinearFunction.LReLU);

There is an example of a "typical" deep network for MNIST with 784 input nodes, and two hidden layers. The first hidden layer has 500 nodes, the next has 100 nodes, and the output layer has 10 nodes. Each layer can have independent nonlinear functions, so don't feel obligated to use the same one for each layer, though that is the most common route.

The NonLinearFunction is a public enum that is provided with the library.

With this, your network is all initialized. Each layer has the matrix of weights and a vector of biases created to the proper dimensions defined in the constructor. These matrices and vectors have been initialized with a value between 0.0 and 1.0 following a normal distribution.

At this point your Network is fully functional, but untrained. You can feed in the values you plan on feeding in after training, but as you'd imagine the results will be more or less random. When you want to run the network, simply use:

_network.FeedForward(double[] input);

This will return the index of the array in which the network places it's "guess" as to where to classify the input.

Training the network

Training has been majorly simplified, just like the Network initialization. All you need to do is construct the NetworkTrainer class with the Network object you wish to train, and call the one exposed function.

Here is a typical NetworkTrainer constructor:

NetworkTrainer _networkTrainer = new NetworkTrainer(_network);

The training function function is called StochasticGradientDescent.

StochasticGradientDescent takes in 6 parameters:

- The number of training data to use each epoch (as an int)
- An array of the training data and labels (as double array Tuples)
- The number of epochs or iterations for which to train (as an int)
- Training mini batch size (as an int - an even divisor of training data count)
- The learning rate (as a double). I recommend starting this around .02-.05
- The testing data (optional - same array type as training data)

This function will yield a tuple whenever an epoch finishes. That is, when it runs through every training image once, adjusts for error, and then runs through the testing data to report how well it is doing. The tuple will be a pair of of an int (the epoch number) with a double (the error rate of that epoch against the testing data).

Hopefully this allows you to get up and running. Putting it all together, you would be able to create and train a network with these steps:

FullyConnectedNetwork _network = new FullyConnectedNetwork ();
_network.AddLayer(784, NonLinearFunction.Sigmoid);
_network.AddLayer(500, NonLinearFunction.Tanh);
_network.AddLayer(100, NonLinearFunction.ReLU);
_network.AddLayer(10, NonLinearFunction.LReLU);
NetworkTrainer _networkTrainer = new NetworkTrainter(_network);
Tuple<double[], double[]>[] _testData = GetTestData();
Tuple<double[], double[]>[] _trainingData = GetTrainingData();
foreach (Tuple<int, double?> _result in _trainer.StochasticGradientDescent(10000, _trainingData, 1000, 100, .05, _testingData))
{
    Console.WriteLine($"Epoch {_result.Item1}: {_result.Item2}%");
}

I hope this is enough to get started. If you have any questions feel free to open an issue in the repo.

Once you get the network trained, you can serialize and store it to keep your training progress. Whenever you deserialize, simply use

_network.FeedForward(double[] newInput);

in order to have your trained network start making predictions.