Simplest artificial neural network
This is the simplest artificial neural network possible explained and demonstrated.
This is part 1 of a series of github repos on neural networks
- part 1 - simplest network (you are here)
- part 2 - backpropagation
- part 3 - backpropagation-continued
Table of Contents
Artificial neural networks are inspired by the brain by having interconnected artificial neurons store patterns and communicate with each other. The simplest form of an artificial neuron has one or multiple inputs each having a specific weight and one output .
At the simplest level, the output is the sum of its inputs times its weights.
A simple example
The idea is to adjust the weights in such a way that the given inputs produce the desired output.
The most common way to measure the error is to use the square difference:
If we had multiple associations of inputs and expected outputs, then the error becomes the sum of each association.
However, in order to adjust the weights of our neural networks for many different inputs and expected outputs, we need a learning algorithm.
The idea is to use the error in order to adjust each weight so that the error is minimized.
What is a gradient?
It's essentially a vector pointing to the direction of the steepest ascent of a function. The gradient is denoted with and is simply the partial derivative of each variable of a function expressed as a vector.
Example for a two variable function:
What is gradient descent?
The descent part simply means using the gradient to find the direction of steepest ascent of our function and then going in the opposite direction by a small amount many times to find the function global minimum.
Gradient descent applied to our example network
Once we have the gradient, we can update our weights
And we repeat this process until the error is minimized within a chosen threshold.
The included example teaches the following dataset to a neural network with two inputs and one output using gradient descent:
How to run
Online on repl.it
docker build -t simplest-network . docker run --rm simplest-network
- Artificial intelligence engines by James V Stone (2019)
- Complete guide on deep learning: http://neuralnetworksanddeeplearning.com/chap2.html