DeepPipe

Deep Learning with elixir. This project is called Deep Pipe(DP). I implemented backpropagation and numerical-gradient. Now I'm testing small data set.

Network example (See test.ex)

defnetwork init_network2(_x) do
  _x |> f(5,5) |> flatten
  |> w(576,100) |> b(100) |> sigmoid
  |> w(100,10) |> b(10) |> sigmoid
end

TODO:

Organize test case

Improve testing

Create a library and register to hex

Installation

sudo apt install build-essential
sudo apt-get install build-essential erlang-dev libatlas-base-dev
mix deps.get

invoking

make clone or download and enter "iex -S mix" on terminal

modules

module DP is Deep Pipe(DP) module module CTensor is code for CNN data operation module Cmatrix is code for Matrix module MNIST is code for MNIST data set module Time is time/1 for measure execution time

running expample

iex(1)> require(Time)
Time
iex(2)> Time.time(Test.momentum(30,100))
preparing data
ready
2.866287227629866
2.5600212240059506
...
0.04318082027257467
0.029026173275906994
0.03131037967594155
0.06550367669302301
accuracy rate = 0.879
"time: 55248299 micro second"
"-------------"
:ok

specification:

data structure

matrix

data

structure of Matrex(CBLAS) e.g. m[2*3]

row_vector

data structure of Matrex(CBLAS) e.g. m[1*3]

tensor

e.g. [m[23],n[23]]

network

e.g. [{:weight,w,lr,v},{:bias,b,lr},{:function,f,g}]

weight

{:weight,w,lr,v} w is matrix, lr is learning rate, v is for momentum,adagrad,adam

bias

{:bias,b,lr,v} b is row vector

filter

{:filter,w,lr,st,v} st is stradd for convolution

pad

{:pad,n} n is size of padding

pool

{:pool,st} st is stradd

function

{:function,f,g,h} f is function, g is differential function, h is function name softmax {:softmax,f,_ } f is function, only output layer softmax is set with cross_entropy

module macros

defnetwork is macros to describe network argument must have under bar to avoid warning message

w(m,n)

weight matrix size(m,n). elements are Gaussian distribution random float

w(m,n,lr)

lr is learning rate (default is 0.1)

w(m,n,lr,z)

z is multiple for Gaussian distribution random float. (default is 0.1)

b(n)

bias row_vector size(n). elements are all zero b(n,lr) lr is learning rate (default is 0.1)

function

sigmoid,relu,ident,softmax

f(m,n)

filter matrix size(m,n). elements are Gaussian distribution random float

f(m,n,lr)

lr is learning rate

f(m,n,lr,z)

z is multiple for Gaussian distribution random float.(default is 0.1)

f(m,n,lr,z,st)

st is stradd

pad(n)

padding n is size of padding

pool(st)

pooling stride size is st

module DP

forward/2

forward calculation for batch data forward(x,network) x is data(matrix or tensor) , network

numerical_gradient/3

calculate gradient by numerical differentiation

numerical_gradient(x,network,t)

x is data, t is train data, loss function is mean_square

numerical_gradient(x,network,t,:cross)

loss function is cross_entropy

numerical_gradient(x,network,t,:square)

loss function is mean_square

gradient/3

caluculate gradient by backpropagation

gradient(x,network,t)

x is data, t is train data

learning/2, /3

update network with gradient

learning(network,gradient)

update with sgd

learning(network,gradient,:sgd)

update with sgd

learning(network,gradient,:momentum)

update with momentum method

learning(network,gradient,:adagrad)

update with adagrad method

learning(network,gradient,:adam)

update with adam method

print/1

print data

newline/0

print LF

save/2

save network data to file

save(filename,network)

load/1

load network data from file

load(filename)

mean_square/2

loss function

mean_square(y,t)

y is row vector of result and t is row vector of train

cross_entropy/2

loss function

cross_entropy(y,t)

y is row vector of result and t is row vector of train

module MNIST

train_image(n)

get train image data size of n. Each data is 28*28 matrix

train_image(n,:flatten)

get train image data size of n. Each data is 784 row_vector

train_label_onehot(n)

get train label data aas onehot row_vector

test_image(n)

get test image data size of n. Each data is 28*28 matrix

test_image(n,:flatten)

get test image data size of n. Each data is 784 row_vector

test_label_onehot(n)

get test label data aas onehot row_vector

module time

time/1

time(func) e.g. time(1+2)