How to Save/Load a NeuralNet via python?

[NKUCodingCat]

I read the interface declared in net/NeuralNet.h and NeuralNet.pyx, seems there is not a function provided to dump an entire NeuralNet with trained weight. However, mentioned weight.dat was not found when I use Python Interface to train network. data from getOutput() in NeuralNet.pyx seems very hard to understand. So is it possible to add such a function which can Save/Load entire CNN ( No matter how it works ) to let a trained network can be used in other place? In a word, how to save and reuse a trained net in Python?

Meanwhile, I simply tried cPickle to dump it but it failed.

[NKUCodingCat]

I tried to hack code to save a net, when I call net.getLayer(0).getWeights(), no matter I train the net or not, I got

Traceback (most recent call last):
  File "W:\Users\admin\Desktop\Reproduce\the_normal_code_2.py", line 66, in <module>
    print(net.getLayer(idx).getWeights())
  File "Layer.pyx", line 40, in PyDeepCL.Layer.getWeights (PyDeepCL.cpp:10189)
IndexError: Out of bounds on buffer access (axis 0)

Alright I found that not all layer have its weights .......

[NKUCodingCat]

When I try to call a RandomTranslation Layer's getOutputCubeSize(), Python crashed immediately, this function defined in Layer Class as base, and try.....except cannot catch this error @hughperkins

[hughperkins]

Where are you seeing RandomTranslationLayer?

DeepCL/python$ grep -i randomtranslate *
grep: benchmarking: Is a directory
grep: build: Is a directory
grep: cmake: Is a directory
grep: DeepCL.egg-info: Is a directory
grep: dist: Is a directory
grep: examples: Is a directory
grep: test: Is a directory

[hughperkins]

well

python$ grep -i randomtranslation *
grep: benchmarking: Is a directory
grep: build: Is a directory
grep: cmake: Is a directory
grep: DeepCL.egg-info: Is a directory
grep: dist: Is a directory
grep: examples: Is a directory
NeuralNet.pyx:                            # used for example by randomtranslations layer (for now,
NeuralNet.pyx:                            # used only by randomtranslations layer)
PyDeepCL.cpp: *                             # used for example by randomtranslations layer (for now,
PyDeepCL.cpp: *                             # used for example by randomtranslations layer (for now,
PyDeepCL.cpp: *                             # used only by randomtranslations layer)
PyDeepCL.cpp: *                             # used for example by randomtranslations layer (for now,
PyDeepCL.cpp: *                             # used only by randomtranslations layer)
PyDeepCL.cpp: *                             # used for example by randomtranslations layer (for now,
PyDeepCL.cpp: *                             # used only by randomtranslations layer)
grep: test: Is a directory

[hughperkins]

Oh, maybe that's the problem, the fact that it isnt defined perhaps?

[hughperkins]

Oh, I see the issue I think. But... how are you creating a random translations layer in order to test this?

[hughperkins]

issue about outputcubsize addressed in 5a53d11 Test output:

DeepCL/python$ py.test -sv test/test_basic.py -k outputcube
============================= test session starts ==============================
platform linux -- Python 3.5.1+, pytest-2.9.2, py-1.4.31, pluggy-0.3.1 -- /norep/envs/env3/bin/python3
cachedir: .cache
rootdir: /data/norep/git/DeepCL/python, inifile: 
collected 5 items 

test/test_basic.py::test_getoutputcubesize X server found.
Using NVIDIA Corporation , OpenCL platform: NVIDIA CUDA
Using OpenCL device: GeForce 940M
statefultimer v0.7
PASSED

===================== 4 tests deselected by '-koutputcube' =====================
==================== 1 passed, 4 deselected in 0.34 seconds ===================

[hughperkins]

getweights() issue addressed in 14716af Also, made it return a numpy tensor (unfortunately 1d for now, but at least its a numpy tensor)

Test output:

DeepCL/python$ py.test -sv test/test_basic.py -k getweights
============================= test session starts ==============================
platform linux -- Python 3.5.1+, pytest-2.9.2, py-1.4.31, pluggy-0.3.1 -- /norep/envs/env3/bin/python3
cachedir: .cache
rootdir: /data/norep/git/DeepCL/python, inifile: 
collected 6 items 

test/test_basic.py::test_getweights X server found.
Using NVIDIA Corporation , OpenCL platform: NVIDIA CUDA
Using OpenCL device: GeForce 940M
statefultimer v0.7
forward try kernel 0
  ... not plausibly optimal, skipping
forward try kernel 1
   ... seems valid
ForwardAuto: kernel 1 0ms
net.getLayer(1).getWeights() None
net.getLayer(2).getWeights().shape (112,)
PASSED

===================== 5 tests deselected by '-kgetweights' =====================
==================== 1 passed, 5 deselected in 0.65 seconds ===================

[hughperkins]

I think pickling/dumping the entire network will take a bit of work. Let's simply save the weights for now, just as you are attempting?

[NKUCodingCat]

The main problem I met is how.can I get the layer's info and reuse it, should I analyze the output of asstring()? Sounds .... uh....complicated

meanwhile, I had not found a wrapper of predict, am I missing something?

[hughperkins]

Bunch of changes in 8fac057

test_setweights test created, and passes:

DeepCL/python$ py.test -sv test/test_basic.py -k test_setweights
============================= test session starts ==============================
platform linux -- Python 3.5.1+, pytest-2.9.2, py-1.4.31, pluggy-0.3.1 -- /norep/envs/env3/bin/python3
cachedir: .cache
rootdir: /data/norep/git/DeepCL/python, inifile: 
collected 7 items 

test/test_basic.py::test_setweights X server found.
Using NVIDIA Corporation , OpenCL platform: NVIDIA CUDA
Using OpenCL device: GeForce 940M
i 2 weightsSize 112
PASSED

================== 6 tests deselected by '-ktest_setweights' ===================
==================== 1 passed, 6 deselected in 0.36 seconds ===================

[hughperkins]

The main problem I met is how.can I get the layer's info and reuse it, should I analyze the output of asstring()? Sounds .... uh....complicated

Hmmm, well, the following methods exist:

• getOutputPlanes()
• getOutputSize()
• getClassName()

Which concrete specific information do you need?

[hughperkins]

meanwhile, I had not found a wrapper of predict, am I missing something?

You can do .forward(images), then call .getLabels() on the last layer:

    net.forward(imagesBatch)
    predictions = net.getLastLayer().getLabels()
    print('predictions', predictions)

There's an example of using this in python/test_lowlevel.py

[NKUCodingCat]

the crash code is something like the net created in test_deepcl_numpy.py, after created a net call net.getLayer(1).getOutputCubeSize(). Do you need my code to test?

Otherwise, what I want to find is a way that I can get all layer's info(necessary info to rebuild a same net) .

[hughperkins]

(for the net weights, there's a c++ class, src/weights/WeightsPersister.h . I can probably wrap that actually. That would allow you to read/write the weights of an entire network. You'd need to know the definiton of that network though)

[hughperkins]

the crash code is something like the net created in test_deepcl_numpy.py, after created a net call net.getLayer(1).getOutputCubeSize(). Do you need my code to test?

I think this is fixed already, in 5a53d11 I guess I should create a new binary release probably right?

[NKUCodingCat]

Alright I will try it later , thanks a lot

[hughperkins]

( v10.0.0 is building now, shoudl be ready in ~15 minutes )

[hughperkins]

(Build in progress:

)

[hughperkins]

v10.0.0 built:

[hughperkins]

Hmmmm, so, I made it so that getOutputCubeSize() throws a normal python exception. But I guess that you probably want it to return the actual output cube size in fact? :-D

[hughperkins]

randomtranslationslayer.getOutputCubeSize fixed in 7e62cb2

DeepCL/python$ py.test -sv test/test_basic.py -k cube
============================= test session starts ==============================
platform linux -- Python 3.5.1+, pytest-2.9.2, py-1.4.31, pluggy-0.3.1 -- /norep/envs/env3/bin/python3
cachedir: .cache
rootdir: /data/norep/git/DeepCL/python, inifile: 
collected 7 items 

test/test_basic.py::test_getoutputcubesize X server found.
Using NVIDIA Corporation , OpenCL platform: NVIDIA CUDA
Using OpenCL device: GeForce 940M
statefultimer v0.7
net.getOutputCubeSize() 75264
PASSED

======================== 6 tests deselected by '-kcube' ========================
==================== 1 passed, 6 deselected in 0.38 seconds ==============

[hughperkins]

(v10.0.1 building)

[NKUCodingCat]

actually I don't know how does this net works, any suggestion about what should I know when I rebuild a net? I am setting a new mobile for my mom so I cannot test anything now.....

[hughperkins]

Well... I guess I'm imagining that you could define the network in a python script, like:

network.py

def create_network(cl):
    ... create network here

Then, when you train, you'll do something like:

import network

... create cl ...
net = network.create_network(cl)
... train network
... save weights to pickle or similar

At prediction:

import network

... create cl ...
net = network.create_network(cl)
... load weights from pickle
... run prediction against new images etc

How does this sound?

[hughperkins]

(v10.0.1 built https://pypi.python.org/pypi/DeepCL/10.0.1 )

[hughperkins]

(note that you'll need to reinstall the native library too, for 10.0.1. This is beause the fix for getOutputCubeSize is actualy in the native underlying library)

[hughperkins]

(the above methodology for save/load network is how it works in torch by the way, eg:

• model definition: https://github.com/szagoruyko/cifar.torch/blob/master/models/vgg_bn_drop.lua
• train: https://github.com/szagoruyko/cifar.torch/blob/master/train.lua
• predict: https://github.com/szagoruyko/cifar.torch/blob/master/example_classify.lua
  )

[NKUCodingCat]

Yeah, I know that......I plan to write a class to handle that, thank you for your tips about it

[NKUCodingCat]

Ok, uh..... I tried to make a class to create network and everything seems great. However, I think that maybe DeepCL can provide an inverse Operation of NetdefToNet.createNetFromNetdef, that is, to generate a netdef string according to a net, so we can just save the InputLayer , the Netdef String and the weights, these can be serialization easier.

But there is a question ..... can each property be represented by Netdef string? I am not sure about it.

[NKUCodingCat]

Uh....... I can not call padZeros(self, bint __padZeros) function defined here https://github.com/hughperkins/DeepCL/blob/master/python/LayerMaker.pyx#L60, is it padZeros(True) the correct way?

[hughperkins]

Hmmmm, you're right, there's a duplicate padZeros(...) method. Fixed in e8ddaf3

Actually, you can simply drop the True parameter, since it defaults to True anyway: it's enough to do .padZeros(), eg see https://github.com/hughperkins/DeepCL/blob/master/python/test_lowlevel.py#L26-L28

[NKUCodingCat]

Yeah, it is.
By the way, what is imageSize means in FullyConnectedLayer? I found that it was set to 1 as default.

[hughperkins]

It doesnt mean anything, you can ignore it. Well... actually... so what it was is, I was using it to predict the next-move in Go, 围棋 , so that would be one of 19x19 output positions. I could represent that as 19x19 = 361 neurons, but I decided it felt more natural to rearrange those into a 19x19 grid.

So, a fully connected layer, it's fully-connected, but you can imagine the outputs as a tower, where the height of the tower is the number of neurons, and each plane in the tower is 1x1. Or you can rearrange that tower into a square grid, and make there be only one such square.

On the whole, unless you have a good reason, you'd probably better keep imageSize to 1, in fully connected layers :-)

[NKUCodingCat]

I think I get the point~ :-P

[NKUCodingCat]

Ok I try to write a script to rebuild a net and it seems work, However, when I try to use this net to predict somthing, it crashed without any error output
codes are here : https://1drv.ms/u/s!AiEbtKTwM8EbhTLN_waVUxz9q3PA

run make_pkl.py to build a seriallized net and run crash_code.py to reproduce the crash, DeepCL_sl.py is the module I write, running in python 2.7

Another question is .... what is your numpy version? maybe it is the reason that you can not use numpy.core.multiarray in pickle/cPickle
hope it can reproduce.

[NKUCodingCat]

I think I had made a same network as the previous one (at least they look same in asString() ) and loaded their weights by using setWeights() function. Technically it can work out a same result as the previous one.

[hughperkins]

When I run make_pkl.py, I get:

Traceback (most recent call last):
  File "make_pkl.py", line 59, in <module>
    128)
  File "NetLearner.pyx", line 4, in PyDeepCL.NetLearner.__cinit__ (PyDeepCL.cpp:16812)
ValueError: Buffer dtype mismatch, expected 'int' but got 'long'

[hughperkins]

So, I changed line 23 and 30, to have , type=np.int32:

labels = np.array(lab, dtype=np.int32)

[hughperkins]

but right, after making that change, it crashes as you are say.

[hughperkins]

Ok, I managed to get this down to the minimum test case to reproduce:

test.py:

import PyDeepCL

def get_net():
    cl = PyDeepCL.DeepCL()
    net = PyDeepCL.NeuralNet(cl, 1, 1)
    net.addLayer(PyDeepCL.ActivationMaker().tanh())
    return net

call_test.py:

import test

net = test.get_net()
net.setBatchSize(1)

Result:

$ python call_test.py
Using NVIDIA Corporation , OpenCL platform: NVIDIA CUDA
Using OpenCL device: GeForce 940M
i 0
i 1
Segmentation fault

[hughperkins]

Ok, moving the cl creation into the caller fixes it. ie:
test.py:

import PyDeepCL

def get_net(cl):
    net = PyDeepCL.NeuralNet(cl, 1, 1)
    net.addLayer(PyDeepCL.ActivationMaker().tanh())
    net.setBatchSize(1)
    return net

call_test.py:

import PyDeepCL
import test

cl = PyDeepCL.DeepCL()
net = test.get_net(cl)
net.setBatchSize(1)

result:

$ python call_test.py
Using NVIDIA Corporation , OpenCL platform: NVIDIA CUDA
Using OpenCL device: GeForce 940M
i 0
i 1
i 0
i 1
$  

(the i 0, i 1, is cos of some debug code I added to DeepCL, you can just ignore those lines)

[NKUCodingCat]

so it means I should pass a cl object into my function instead of create in it?
is it caused by python's gc behaviour?

[hughperkins]

so it means I should pass a cl object into my function instead of create in it?

That will get your code working now, yes.

is it caused by python's gc behaviour?

Yes, I assume so. Whether I can, eg assign the cl object to the net object, eg in net's __init__ method is something I'm pondering.

[hughperkins]

Addressed in 450dba1 I'll create a new binary soonish (I'll probably wait till the next hour, to start the ec2 build instances).

[NKUCodingCat]

I try to return cl object with net object and it solved the crash too , therefore it must be a gc-caused problem. Easy to cause a bug, I think.

[NKUCodingCat]

Oh I have another request, can I disable the output of cl infos? something like

   forward kernel 4 time: 1ms
   forward kernel 5: cannot be used
   forward kernel 6 time: 3ms
   forward kernel 7 time: 706ms
   forward layer selected kernel 4
forward try kernel 5
ForwardAuto: kernel 5: this instance cant be used: For ForwardFc, filtersize and inputimagesize must be identical
   ... not valid
forward try kernel 6
   ... seems valid
ForwardAuto: kernel 6 3ms
forward try kernel 5
cl/forward_fc_wgperrow.cl build log:
"W:\Users\admin\AppData\Local\Temp\OCL21BD.tmp.cl", line 75: warning: variable
          "loopsPerExample" was declared but never referenced
      const int loopsPerExample = (gInputSize + workgroupSize - 1) / workgroupSize;
                ^


   ... seems valid
ForwardAuto: kernel 5 2ms
forward try kernel 5
cl/forward_fc_wgperrow.cl build log:
"W:\Users\admin\AppData\Local\Temp\OCL221B.tmp.cl", line 75: warning: variable
          "loopsPerExample" was declared but never referenced
      const int loopsPerExample = (gInputSize + workgroupSize - 1) / workgroupSize;
                ^


   ... seems valid
ForwardAuto: kernel 5 3ms

[hughperkins]

You mean, disable all the spammy stuff, that you've shown in the codeblock?

[NKUCodingCat]

yeah, is it possible to easily disable them? Though it is harmless but it is not useful for using an network, either.

[NKUCodingCat]

just like the verbose option when booting the hackintosh, if it does not work ,we need it, but they were hidden in default :-P

[hughperkins]

Hmmm, seesm like configurable logging will need a bit more work than exposing one switch. Will ponder...

[hughperkins]

Seems like this would need some kind of configurable logging framework. I'm not really up on which logging frameworks work well in c++. thoughts?

[NKUCodingCat]

Uh...no idea about that...cpp is unfamiliar for me...

seems most of cpp program do not need a logging framework...

[hughperkins]

seems most of cpp program do not need a logging framework...

Not sure. Basically, in our case we want to be able to turn logging on/off at runtime. So, at the very least would need to create some methods like debug, info, warning etc, and replace all the couts with those. That's ... quite a lot of work :-P

[NKUCodingCat]

how about google glog
just replace cout with LOG(INFO) seems easy to use(just looks easy and I had not tried it)

[viper7882]

Hi @hughperkins,

I'm looking for a way to save and load the trained net in Python 2.7. Looking at the changes made by @NKUCodingCat seems logical. However I'm unable to find any of his save and load functionality available in the examples provided by DeepCL. By any chance if you have plan to provide public Python API for saving and loading the net in Python 2.7?

[hughperkins]

I do not intend to do that. But just through lack of time, rather than any fundamental objection. Is this something you might consider contributing to?

[viper7882]

Noted with thanks. Let me see how I could contribute.