Python optimizer.py example optimizer

[oysteijo]

The optimizer.py code is just an example of how the optimizer may be in a C implementation.
It is indeed work in progress, and this fiel might be deleted in the future.

As an example, here is an interactive session I had with the code:

In [1]: import numpy as np
In [2]: from optimizer import SGD
In [3]: from reference import NeuralNet
In [4]: nn = NeuralNet("../optimizer/initial_weights.npz", "sigmoid, sigmoid", loss="mean_squared_error" )
In [5]: opt = SGD( nn, learning_rate = 0.01 )
In [6]: arr = np.load("../optimizer/contact_training_and_test.npz")
In [7]: train_X, train_Y, val_X, val_Y = tuple((arr[m] for m in arr))
In [8]: for e in range(20):
   ...:     opt.run_epoch(  train_X, train_Y, val_X, val_Y  )
 mean squared error valid:  0.022117715
 mean squared error valid:  0.016992321
 mean squared error valid:  0.01156354
 mean squared error valid:  0.008583443
 mean squared error valid:  0.00779693
 mean squared error valid:  0.0074062655
 mean squared error valid:  0.0071243667
 mean squared error valid:  0.0069057867
 mean squared error valid:  0.006728847
 mean squared error valid:  0.0065795756

... and maybe needless to say, this i considerably slower than the C implementation.