DINRHIW2 Machine Learning Library
Developed between 2002-2004, 2013-2023. Picture: GB-RBM reconstructed data (red) from examples (green). Variance/probability distribution of the input data is approximated more or less correctly.
Machine learning component contains:
- unix commands line tools to do machine learning from scripts (nntool, dstool)
- C++ implementation (CPU) means that the library should be better than Python or Java in real-time and stand-alone applications (binaries for regular/older laptops without fast GPU)
-
deep learning works reasonable well with nntool (test_titanic.sh and leaky ReLU non-linearity,
test_data_residual.shuses a 20-40 layer residual neural network [20 layers work perfectly with a simple test problem, 40 layers still give reasonable results]). - finally implemented popular ADAM gradient descent optimizer which is FAST and gives often MUCH BETTER results..
- There is partial support for complex valued neural networks. I have also implemented sgrad optimizer (nntool) and PCA/ICA codes which uses my experimental number system where circular convolution is number multiplication (see in the docs my research article about it, results are currently slightly worse with my superresolutional numbers).
- multistart L-BFGS neural network 2nd order optimizer
- drop out heuristics for deep learning code
- basic BayesianNeuralNetwork using hybrid monte carlo sampling
- principal component analysis (PCA) preprocessing method including symmetric eigenvalue decomposition code (eig) and linear ICA
- Hidden Markov Model (HMM) learning
- Variational Autoencoder code (VAE.h / VAE.cpp)
- [EXPERIMENTAL CODE]: restricted boltzmann machines (RBM): GB-RBM & BB-RBM. (DBN.h / GBRBM.h / BBRBM.h)
- [EXPERIMENTAL CODE]: basic reinforcement learning code (discrete/continuous actions, continuous states)
- [EXPERIMENTAL CODE]: parallel tempering sampling
- recurrent neural network code and code using BPTT algorithm
- multi-threaded (multicore CPUs) code to support parallel search/optimization (std::threads and OpenMP)
- uses BLAS interface for matrix/vector math which is optimized for speed (OpenBLAS, Intel MKL, AMD BLIS). There is also experimental NVIDIA CUDA support but it is currently slow because of constant synchronizations between GPU RAM and PC's RAM which should be optimized.
- supports CSV ascii file import/export
- t-SNE dimension reduction, SOM self-organizing map (slow), simplex optimization
- 4th order Runge-Kutta numerical differential equation solver
- quaternion, 3d matrix rotation and other codes
Additionally, the library contains basic linear algebra and various other general computer science algorithms like radix sorting (for floating point numbers through conversion code O(N)), hermite curve interpolation and cryptography (AES, DES, Triple-DES, RSA..), arbitrary precision numbers (HMM requires arbitrary precision numbers) etc. The compiler used is GCC https://gcc.gnu.org/ and the build process uses GNU Makefile and autoconf. The library also uses C++20 features meaning that only a new GCC C++ compiler is likely to work. The project also compiles using Clang/LLVM compiler but is not fully tested using Clang.
Since April 2017 there are ~100.000 lines of C++ code in the library.
Compiles both on Linux and on Windows (GCC/Linux, MSYS2/MINGW).
Releases are distributed using GNU GPL license. Links with BSD/LGPL libraries ( gmp, blas and python (python dependency can be removed)).
Author (Tomas Ukkonen) keeps full rights to use library in anyway he wants in the future including releases under different licenses.
Implement more modern algorithms (C++):
- add cuBLAS support for accelerated BLAS routines (current experimental CUDA implementation is very slow (continuous data transfers between CPU and GPU))
- convolutional (sparse) neural networks
- LSTM RNNs
- tranformer neural networks+attention mechanism
class nnetwork<>
class dataset<>
class NNGradDescent<>
class LBFGS<>
class LBFGS_nnetwork<> (neural network LBFGS optimizer)
class HMC<> (parallel hamiltonian monte carlo sampling for neural networks)
tools/nntool
tools/dstool
are the primary classes and tools to look at and use. Also read HowToLearnFromData page. Those interested in pretraining weights using RBMs, read docs/RBM_notes.tm and src/neuralnetwork/tst/test.cpp: rbm_test() and look at classes:
class GBRBM<>
class BBRBM<>
class DBN<>
It is possible to convert layer-by-layer optimized DBN into feedforward supervised neural network.
For mathematical documentation, read: RBM notes, bayesian neural network notes and recurrent neural network gradient notes.
Recurrent neural networks are activated using --recurrent DEPTH in nntool. For more information read:
class SGD_recurrent_nnetwork.h
which trains basic recurrent nnetwork using LBFGS. For examples, look at src/neuralnetwork/tst/test.cpp.
- OpenMP uses too many threads which is often slower than single threaded code. (FIX: use "OMP_NUM_THREADS=1" or "OMP_SCHEDULE=dynamic,25" env variables)
- cleanup the code and remove classes that are not needed and don't belong to this library
- try to remove few direct x86 dependencies (cpuid instruction)
- PROPER DOCUMENTATION (these wiki pages are also slightly out of date)
- small things: write bash autocomplete scripts for nntool and dstool and make nntool work better in command-line
The software has been developed on Linux (x86 and amd64). It also compiles using MINGW-w64 64bit/Windows compiler (MSYS2 environment). It has been reported to work correctly in Cygwin 32bit environment (but cygwin license is a problem). Currently there are few x86 specific instructions (CPUID) and assumptions (size of int and float etc) so only x86 and amd64/x86_64 are fully supported.
Tomas Ukkonen tomas.ukkonen@iki.fi
GitHub is now (2015) the official location of this project.