Tomas Ukkonen edited this page Sep 20, 2018 · 78 revisions

Developed between 2002-2004, 2013-2018. Picture: GB-RBM reconstructed data (red) from examples (green). Variance of input data is approximated more or less correctly.

GB-RBM reconstructed data (red) from examples (green)

Machine learning component contains:

  • unix commands line tools to do machine learning from scripts (nntool, dstool)
  • multistart L-BFGS neural network 2nd order optimizer
  • basic BayesianNeuralNetwork using hybrid monte carlo sampling
  • principal component analysis (PCA) preprocessing method including symmetric eigenvalue decomposition code (eig)
  • basic Hidden Markov Model (HMM) codes
  • Stacked RBMs (GB-RBM & BB-RBM)
  • [EXPERIMENTAL CODE]: basic reinforcement learning code (discrete/continuous actions, continuous states)
  • [EXPERIMENTAL CODE]: parallel tempering sampling
  • recurrent neural network code using BB-RBM output (RNN-RBM) and extended RTRL algorithm
  • multi-threaded (multicore CPUs) code to support parallel search/optimization (std::threads and OpenMP)
  • uses BLAS interface for matrix/vector math which is optimized for speed (OpenBLAS, Intel MKL)
  • supports CSV ascii file import/export
  • self-organizing map, quaternion, 3d matrix rotation and other codes

Additionally, the library contains basic linear algebra and various other general computer science algorithms like radix sorting (for floating point numbers through conversion code O(N)), hermite curve interpolation and cryptography (AES, DES..), arbitrary precision numbers C++ library etc. The compiler used is GCC https://gcc.gnu.org/ and the build process uses GNU Makefile and autoconf. The library also uses C++11 features meaning that only a new GCC C++ compiler is likely to work.

As of April 2017 there is ~100.000 lines of C++ code in the library.

Compiles both on Linux and on Windows (GCC).

NOTE: Please use experimental branch (RBM_test) if possible as the main branch is not actively maintained and I only plan to drop updates to it from experimental branch from time to time.

LICENSE

Releases are distributed using GNU GPL license. Links with BSD/LGPL libraries (zlib, gmp and blas).

Commercial license is also available. C++ implementation allows integration of machine learning into native C++ applications. Contact author(s) for details. Author(s) (Tomas Ukkonen) keep full rights to use library in any way he or she wants in the future
including releases under different licenses.

PLAN

Implement more modern algorithms (C++):

  • add cuBLAS support for accelerated BLAS routines
  • convolutional (sparse) neural networks
  • LSTM RNNs

DOCUMENTATION

class nnetwork<>

class dataset<>

class LBFGS<>

class LBFGS_nnetwork<> (neural network LBFGS optimizer)

class HMC<> (parallel hamiltonian monte carlo sampling for neural networks)

tools/nntool

tools/dstool

are the primary classes and tools to look at and use. Also read HowToLearnFromData page. Those interested in pretraining weights using RBMs, read docs/RBM_notes.tm and src/neuralnetwork/tst/test.cpp: rbm_test() and look at classes:

class GBRBM<>

class BBRBM<>

class DBN<>

It is possible to convert layer-by-layer optimized DBN into feedforward supervised neural network.

For mathematical documentation, read: RBM notes, bayesian neural network notes and recurrent neural network gradient notes.

Recurrent neural networks are activated using --recurrent DEPTH and --pseudolinear switches in nntool. For more information read:

class rLBFGS_nnetwork<>

which trains a primitive recurrent nnetwork using LBFGS. For examples, look at src/neuralnetwork/tst/test.cpp.

TODO

  • cleanup the code and remove classes that are not needed and don't belong to this library
  • improve threading code: fix known minor race conditions in parallel code
  • try to remove few direct x86 dependencies (cpuid instruction)
  • PROPER DOCUMENTATION (these wiki pages are also slightly out of date)
  • small things: write bash autocomplete scripts for nntool and dstool

PLATFORMS

The software has been developed on Linux (x86 and amd64). It also compiles using MINGW-w64 64bit/Windows compiler (MSYS2 environment). It has been reported to work correctly in Cygwin 32bit environment (but cygwin license is a problem). Currently there are few x86 specific instructions (CPUID) and assumptions (size of int and float etc) so only x86 and amd64/x86_64 are fully supported.

CONTACT DETAILS

Tomas Ukkonen tomas.ukkonen@iki.fi

GitHub is now (2015) the official location of this project.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.