Low level C++ neural network engine. The engine provides a huge flexibility in creating neural networks.
Even more flexibility were provided in v2 comparing to previous version!
- Table of Contest
- Build
- Documentation
- Definitions
- Types and aliases
- class nerone::value::Value
- class nerone::value::FValue
- class nerone::value::GValue
- class nerone::value::Cumulative
- nerone::cell::Cell
- nerone::cell::VCell
- nerone::cell::OCell
- nerone::cell::TCell
- nerone::cell::Functional<VT, FN>
- Functional()
- Functional(shared_value_t input, shared_value_t output)
- Functional(FN act, FN grad)
- Functional(shared_value_t input, shared_value_t output, FN act, FN grad)
- Functional(shared_value_t input, FN act, FN grad)
- FN get_act_fn()
- FN get_grad_fn()
- void set_act_fn(FN)
- void set_grad_fn(FN)
- const value_list_t& get_inputs()
- const value_list_t& get_outputs()
- void set_inputs(value_list_t vals)
- void set_outputs(value_list_t vals)
- void set_input(shared_value_t value)
- void set_output(shared_value_t value)
- void clean()
- nerone::cell::Activation
- nerone::cell::Loss
- nerone::cell::Segment
- Segment(value_list_t inputs, value_list_t outputs)
- const value_list_t& get_inputs()
- const value_list_t& get_outputs()
- void set_inputs(value_list_t vals)
- void set_outputs(value_list_t vals)
- void calc_value()
- void calc_grad()
- void clean()
- const weight_list_t& get_weights()
- void set_weights(weight_list_t weights)
- void update(shared_train_data_t data)
- nerone::cell::Cluster
- Cluster(cell_list_t cells)
- void update(shared_train_data_t data)
- void calc_value()
- void calc_grad()
- void clean()
- void start()
- void finish()
- void command(shared_cell_command_t com)
- const weight_list_t& get_weights()
- void set_weights(weight_list_t weights)
- virtual const cell_list_t& get_cells()
- virtual void set_cells(cell_list_t cells)
- nerone::cell::Layer
- nerone::cell::Sequense
- nerone::cell::LossLayer
- nerone::Producer
- nerone::GDProducer
- Activation functions
- Loss functions
- Usage example
- Tests
- Additional information
- License
Library was tested using GNU G++ compiler with flag -std=c++17. So it is recommended to use C++ 17 or higher version of compiler. Compiling by other compilers may require code corrections.
make default command builds the tests and creates test.exe file in the root directory.
- Value - is a structure that is used as inputs and outputs of cells. It contains input/output related data, e.g. floating point cell output.
- Cell - is a structure that has some (maybe zero) input and output values, and provides a way for manipulating these values.
- Cluster - is a structure containing multiple cells, and ruling these cells.
- Producer - is a structure that provides algorithms for managing and ruling clusters.
Current engine provides nerone namespace where all the classes, types and functions are defined.
- nerone::shared_value_t - alias for std::shared_ptr<nerone::value::Value>
- nerone::shared_cell_t - alias for std::shared_ptr<nerone::cell::Cell>
- nerone::value_list_t - alias for std::vector<nerone::shared_value_t>
- nerone::cell_list_t - alias for std::vector<nerone::shared_cell_t>
- nerone::act_fn_t - alias for template<typename FT> std::function<FT(FT)>
- nerone::loss_fn_t - alias for template<typename FT> std::function<FT(FT,FT)>
- nerone::shared_train_data_t - alias for std::shared_ptr<nerone::TrainData>
- nerone::shared_cell_command_t - alias for std::shared_ptr<CellCommand>
Basic abstract value class with default constructor and default virtual destructor.
It doesn't provide any other functions or properties.
Abstract value class that extends nerone::value::Value with default constructor and destructor and provides 2 pure virtual methods and public type f_type = T.
Main purpose of this abstraction is to define a value floating point type, and provide get/set value interface.
Pure virtual method for getting the floating point value.
Pure virtual method for setting the floating point value.
Abstract value class that extends nerone::value::FValue<T> with default constructor/destructor. It provides additional 2 pure virtual methods.
Main purpose of this abstraction is to provide get_grad/set_grad interface for manipulation value's gradient floaing point values.
Pure virtual method for getting the floating point gradient value.
Pure virtual method for setting the floating point gradient value.
Cumulative value class. It extends nerone::value::GValue It defines public f_type = T type.
This value class uses a stack for storing it's values.
It implements all pure virtual methods, and adds a couple of more methods for manipulating the memorty of the value.
Returns the last value from the stack.
Sets the new value to be on top of the value stack.
Returns the grad value.
Sets the grad value, and pops the most recently set primary value from the stack.
Removes the top value from the stack.
Cleans up the stack.
Returns true if the value stack is empty.
Examples:
auto val = std::make_shared<nerone::value::Cumulative<float>>();
val->set(0.1);
val->get(); // 0.1
val->set(0.2);
val->get(); // 0.2
val->set_grad(0.5);
val->get(); // 0.1, as set_grad method removes the top value.
val->set_grad(0.6);
val->get(); // 0.1, the stack is empty, but the value is still remembered unless its overwritten by new `set()` call.Note: for examples on how it all works together, please reach to the Cumulative value tests.
Basic abstract cell class with default constructor and default virtual destructor.
It also defines 4 pure virtual methods:
Returns the list of input value.
Returns the list of output values.
Assigns input values.
Assigns output values.
Abstract template class that extents nerone::cell::Cell with default constructor, and defines a single public value_type = VT.
The main purpose of this class is to define the value type that is used with the cell.
Note: each class that extends nerone::cell::VCell class should define public value_type.
Abstract Operationl class that extends nerone::cell::VCell<VT> and additionally defines next pure abstract methods:
Note: this cell accepts VT type that extends nerone::value::Cumulative.
The method should calculate and set the cells output primary values.
The method should calculate and set the cells input gradient values.
Cleans up values.
Allows sending additional operational signals to cells. Can be overwritten and special meaning set for specific command.
By default the function doesn't do anything whatever command is sent.
Prepares the values or any other data for calc methods to be called. It doesn't do anything fir this specific class, but can be overwritten and the start meaning defined.
Cleaups the values or any other data. Basically it should return the cell into it's original state.
Abstract trainable class that provides trainable interface. It extends nerone::cell::OCell<VT>
It also defines public weight_list_t = std::vector<VT::f_type> type, and next trainable pure virtual methods:
Returns the list of weight floating values.
Assigns new weights to the current trainable cell.
Should update the weights with new floating values. It might take data sent as an argument into consideration. It may contain learning rate for example, etc.
The abstract cell extends operational class (nerone::cell::OCell<VT>). It accepts 2 template parameters - VT is for value type, and FN is for activation/gradient function type. It has 1 input and 1 output, and provides a banch of constructors to fill in all of the internal states.
Default constructor that assigns empty input and output values to the cell values, and assigns nerone::activation::Linear activation and gradients functions to the cell activation and gradient functions respectifully.
Assigns input value and output value to the cells input and output values respectifully, assigns linear activation and gradients functions.
Assigns passed functions act and grad to the cells activation and gradient functions, and creates and assigns empty input and output values.
Assigns input and output to cells input and output, assigns act and grad to cells activation and gradient functions.
Like previous one, but only input value is assigned to cells input value, and output value is created assigned as empty value.
Returns activation function.
Returns gradient function.
Assigns new activation function.
Assigns new gradient function.
Returns the list of inputs with exactly one input value in it.
Returns the list of outputs with exactly one output value in it.
Sets up new input values. The list should contain only one value.
Sets up new outputs values. The list shlould contain only one value.
Sets up input value.
Sets up output value.
Cleanups input and output values.
The class extends nerone::cell::Functional<VT, nerone::act_fn_t<VT::f_type>>. It accepts one template parameter VT - value type.
It inherits all parent class constructors, and implements the last 2 pure virtual methods:
Applies activation function to the input value, and set the result to the output value.
Applies gradient function to the output gradient value, and set the result to the input gradient value.
The class extends nerone::cell::Functional<VT, nerone::loss_fn_t<VT::f_type>>. It accepts one template parameter VT - value type.
It inherits all parent class constructors, implements the last 2 pure virtual methods, and adds couple of more methods:
Sets the expected input value to the passed expected floating value, and runs void calc_value().
Gets calculated value from its input value, and expected value from its expected input value, and uses them as inputs to its loss activation function. The result then assigned to the cells output value. This is basically the error value.
Gets calculated value from its input value, and expected value from its expected input value, and uses them as inputs to its loss grad function. The result then assigned to the cells input grad value.
Cleans up input, output and expected values.
Assigns expected input to the cell.
Returns currently assigned expected input.
The cell extends nerone::cell::TCell<VT> and implements all of the parents classes abstract methods.
It accepts one template parameter VT - value type.
The main purpose of this cell is to create, keep and manage dense connection (weights) between input and input values.
It also defines 3 public types:
- value_type = VT
- weight_list_t = TCell<VT>::weight_list_t
- f_type = VT::f_type
There is default constructor exists.
Assigns passed inuts and outputs to cells inputs an outputs respectifully, and creates the matrix of weights.
Returns the list of input values.
Returns the list of outputs values.
Assigns new list of input values, and retuilds the weights matrix in case of the new input size is different from the old input size.
Same as above, but for output values.
Calculates and sets output values floating values by the next formula:
Output[i] = SUM{j = [0, INPUT_SIZE]}(Input[j] * weights[i][j])
Calculates and sets input values floating gradient values.
Cleanups inputs and outputs values and internal weights gradients.
Note: weights are not cleaned by calling this method.
Returns the list of weights.
Assigns new list of weights to the cell.
Note: the amount of weihts should be qual to inputs.size() * outputs.size()
Accepts GDTrainData smart pointer as an input, and uses it's learning_rate attribute to update the weights.
Note, each call to calc_grad cumulates the gradient for each weight, and when update function is called, its uses these cumulated values.
*That's how the batch size is simulated.
Abstract class that extends nerone::cell::TCell<VT>. It accepts one template parameter VT - value type.
The main purpose of this cell is to keep and manage other cells. It doesn't really keep weights or it's own inputs, ouput or other values, so it inplements some of the virtual methods.
It also has a default constructor
Note: all cells that cluster keeps must have Operational as one of their parents.
Constructor that accepts the list of cells.
Basically just runs update method to all of its Trainable cells.
Calls calc_value method for all of itscells.
Calls calc_grad method for all of its cells.
Calls clean method for all of its cells.
Calls start method for all of its cell.
Calls finish method for all of its cells.
Calls command method for all of its cells passing the same argument.
returns the empty list.
Doesn't really do anything.
Returns the list of this cluster cells.
Assigns new list of cells to the cluster.
Cluster cell that extends nerone::cell::Cluster<VT> and keeps cells that needs to be executed in paralel, making its inputs to be all the inputs of all its cells, and its outputs to be all the outputs from all of its cells.
It accepts one template type VT - value type.
Has default constructor.
Constructor that accepts and assigns cells that cluster will keep.
returns the list of inputs of all its cells.
returns the list of outputs of all its cells.
Assigns new input values to all of its cells.
Assigns new output values to all of its cells.
Cluster cell that extends nerone::cell::Cluster<VT>. It has one template type VT - value type.
The porpose of this cell is to collect other operational cells, and apply methods to them in a sequentional manner.
Default constructable.
Constructor that accepts and assigns cells that cluster will keep.
Returns its first cells inputs.
Returns its last cells outputs.
Assigns value lists to its first cell inputs.
Assigns value list to its last cell outputs.
Automatically connects all its cells inputs to outputs. Basically it assigns each cell outputs to next cell inputs.
The sequense cell that extends nerone::cell::Layer<VT>, nerone::cell::LossClusterInterface.
The porpose of this cell is to collect other operational cells, and apply them in a sequentional manner.
Constructor that accepts the list of cells.
returns the list of expected inputs from all of the cells.
Accepts the list of values and assigns them to expected values on each cell.
Class that conducts all of the procesess related to training the neural network, and executing queries.
Each producer contain one body and one loss cells to operate with.
It defines 2 public types:
- body_cell_t = std::shared_ptr<cell::Cluster<VT>>
- loss_cell_t = std::shared_ptr<cell::LossLayer<VT>>
It has default constructor and default virtual destructor.
Constructor that accepts body cell and loss cells.
Assigns body cell.
Assigns loss cell.
Returns its body cell.
Returns its loss cell.
There is a namespace nerone::activation with a bunch of activation functions already defined. Each of them consist of a class with 2 public static members fun and grad - activation function itself and a derivative of the activation function.
Each member is a template function with one template paraneter T that requires floating point type to be passed. It accepts one argument with type T, and returns T.
fun function returns the result of applying activation function to the passed value. grad function returns the result of applying gradient function to the passed value.
All activation functions described below live under the nerone::activation namespace.
The sigmoid function is a logistic function, which means that, whatever you input, you get an output ranging between 0 and 1.
Parametric Reflicter activation function. For values less than 0, the result equals to value param. For values greater than 0, the result doesn't change. R should implement std::ratio class.
Example:
// Define class with parameter = 1/10 = 0.1
using ReLU01 = nerone::activations::ReLU<std::ratio<1,10>>
ReLU01::fun<float>(-10.0) // 1
ReLU01::fun<float>(15.0) // 15The output of the tanh activation function is Zero centered; hence we can easily map the output values as strongly negative, neutral, or strongly positive. Usually used in hidden layers of a neural network.
Parametric Exponential Linear Unit activation function. For values less than 0, the result goes to -1 softly. Parameter decides the curve line for negative values. For values greater than 0, the result doesn't change. R should implement std::ratio class.
Gaussian Error Linear Unit activation function. For values much less than 0, the result equals 0, but for values slightly less than 0, the result slightly less than 0. For values greater than 0 the result doesn't change.
Scaled Exponential Linear Unit. This activation functions is one of the newer one's.
Swish is a smooth, non-monotonic function that consistently matches or outperforms ReLU on deep networks applied to a variety of challenging domains such as Image classification and Machine translation.
There is a nerone::loss namespace with 2 loss functions already defined. Each of them consist of a class with 2 public static members fun and grad - activation function itself and a derivative of the activation function.
Each member is a template function with one template paraneter T that requires floating point type to be passed. It accepts two argument with type T (actual value, and expected value), and returns T.
fun function returns the result of applying loss function to the passed values. grad function returns the result of applying gradient loss function to the passed values.
All loss functions described below live under the nerone::loss namespace.
Mean squared error (MSE) is the most commonly used loss function for regression. The loss is the mean overseen data of the squared differences between true and predicted values.
Cross-entropy loss, or log loss, measures the performance of a classification model whose output is a probability value between 0 and 1. Cross-entropy loss increases as the predicted probability diverges from the actual label.
There is a ner_one.h file that includes all other nerone header files.
#include <iostream>
#include <memory>
#include <cstdlib>
#include "ner_one.h"
#include "src/build_helpers.cpp"
using ExecutionCase = nerone::GDProducer<nerone::value::Cumulative<float>>::ExecutionCase;
using ExecutionResult = nerone::GDProducer<nerone::value::Cumulative<float>>::ExecutionResult;
const double PI = 3.1415;
float rand_double() {
return (rand()%10000)/(float)10000;
}
// Structure defining point on a plane, and bool parameter defining
// if the point belongs to red team or to blue team (true and false)
struct Point {
double x;
double y;
bool red;
};
// Creates set of points devided for 2 sections, and puts red points
// to the top-right area of the plane, and blue points to the
// bottom-right are of the plane.
std::vector<Point> create_labeled_inputs_zones_2() {
std::vector<Point> res;
// top-right zone
for(int i=0;i<100;i++){
double len = rand_double() * 15;
double a = rand_double() * PI * 2;
res.push_back({len * std::sin(a) + 65, len * std::cos(a) + 35, true});
}
// bottom-left zone
for(int i=0;i<100;i++){
double len = rand_double() * 15;
double a = rand_double() * PI * 2;
res.push_back({len * std::sin(a) + 35, len * std::cos(a) + 65, false});
}
return res;
}
// Normalize the data
float norm(float a) {
return a / 100 * 2 - 1;
}
void print_tests_result(std::vector<ExecutionCase> tests, std::vector<ExecutionResult> res) {
for(int i=0;i<(int)tests.size();i++) {
std::cout << tests[i].values[0] << " " << tests[i].values[1] << " " << res[i].values[0] << " " << res[i].values[1] << std::endl;
}
}
int main() {
srand(12345);
// Creates a producer.
auto *gd = new nerone::GDProducer<nerone::value::Cumulative<float>>();
std::vector<int> layers{2, 6, 6, 2};
// Prepares the weights matrix initial values.
std::vector<std::vector<float>> weights;
for(int i=1;i<(int)layers.size();i++) {
int cnt = (layers[i-1]+1) * layers[i];
std::vector<float> layer;
for(int j=0;j<cnt;j++) {
layer.push_back(rand_double());
}
weights.push_back(layer);
}
// Builds the cluster.
ConnectedLayerBuilder b(true);
b.add(layers[0], nerone::activation::Linear::fun<float>, nerone::activation::Linear::grad<float>);
for(int i=1;i<(int)layers.size();i++) {
b.add(layers[i], nerone::activation::Sigmoid::fun<float>, nerone::activation::Sigmoid::grad<float>);
}
b.withLoss(nerone::loss::CrossEntropy::fun<float>, nerone::loss::CrossEntropy::grad<float>);
b.withWeights(weights);
auto [body, loss] = b.build();
// Assigns the cells.
gd->set_body_cell(body);
gd->set_loss_cell(loss);
// Configure the producer.
gd->set_learning_rate(0.01);
gd->set_batch_size(1);
gd->set_randomized(true);
auto points = create_labeled_inputs_zones_2();
std::vector<ExecutionCase> cases;
for(auto &p : points) {
cases.push_back({{norm(p.x), norm(p.y)}, {p.red ? 1.0f : 0.0f, p.red ? 0.0f : 1.0f}});
}
// Trains on labeld data.
for(int i=0;i<100;i++) {
gd->train(cases);
}
std::vector<ExecutionCase> tests;
for(int i=0;i<100;i++) {
for(int j=0;j<100;j++) {
tests.push_back({
{norm(j), norm(i)},
{}
});
}
}
// Executes test data.
auto res = gd->execute(tests);
// print_tests_result(tests, res);
return 0;
}Regular unit tests are placed under the tests folder.
Using the example I build above, with logging of additional useful data such as error function, and tests results I've been able to train neural network for different classification problems:
-
training step: 20 | error: 0.470739
-
training step: 40 | error: 0.0616046
-
training step: 80 | error: 0.0130729
-
training step: 100 | error: 0.00906824
-
training step: 30 | error: 0.626602
-
training step: 120 | error: 0.114609
-
training step: 240 | error: 0.029581
-
training step: 300 | error: 0.0197742
-
training step: 80 | error: 0.678708
-
training step: 160 | error: 0.617172
-
training step: 280 | error: 0.371635
-
training step: 400 | error: 0.0287993
-
training step: 1000 | error: 0.662982
-
training step: 4000 | error: 0.479241
-
training step: 7000 | error: 0.133034
-
training step: 10000 | error: 0.0362706
As we can see, the neural network is able to be trained to solve these classification problems.
This project is a low level neural network engine. It is not a complete neural network framework you can use for real world tasks. To be so, additional wrapper required to be built on top of this engine providing methods to easier build of neural network, and even more important, reliable GPU optimized cells needs to be implemented to improve neural network training performance.
MIT
Have fun :)