convert the model of objection

[wangcm12]

Hi Bodo,,
Your snn_toolbox is very good. I am learning SNN. I am learning your code, I want to convert the model of objection, for example ,YOLO, I found your model is about classification，I tried convert the YOLO (it is Keras model), but failed. I read your documentation ,but I havn't idea. What should I do? Should I going to write a function about detecting positioning? Can you give me suggestion?

[rbodo]

Hi,

Thanks for your question. Implementing spiking networks for object detection would be a really cool feature for the toolbox. It will need some work though (and I won't be able to do it myself).

As you say, the toolbox expects classifiers, so whenever the network is evaluated (before and after parsing, and after conversion), the toolbox tries to compute an accuracy score by taking the argmax of the network output and comparing it against the provided labels (in the file y_test.npz). This is the part that you will have to change. Instead of providing just one class label per image, you will have to provide (possibly multiple) labels and bounding boxes for each frame. And you will have to tell the toolbox not to compute accuracies, but mean-average-precision scores.

Another thing that you need to check is that the logging and plotting functions do not break. The toolbox can be configured to write various quantities to disk during and after simulation, and to generate plots of intermediate results etc. A simple workaround would be to just let the toolbox store and plot zeros for the accuracies and other things that you removed. But you may want to add functionality to store and plot your new output so you can debug and evaluate more easily.

I suggest you try to follow the processing pipeline here to see what happens to the output of the network after simulating the SNN:

snn_toolbox/snntoolbox/simulation/target_simulators/INI_temporal_mean_rate_target_sim.py

Line 151 in 5f85df1

out_spikes = self.snn.predict_on_batch(input_b_l)

Basically everything after this line is about decoding the network output, logging and plotting results. This is where you will have to modify most.

Here is one of the places where the ANN get's evaluated, which will have to be modified as well:

snn_toolbox/snntoolbox/parsing/model_libs/keras_input_lib.py

Line 165 in 5f85df1

score = val_fn(x_test, y_test, batch_size, verbose=0)

Good luck!

[wangcm12]

Thanks!
I found the h5 file of vgg not in folder. I used the model trained by my own data set， but it reported a error"can not found the model", I edit iconfig according to the given example，any suggestion?

[rbodo]

The h5 file of VGG is too big to put in the repository.

You need to set the parameter filename_ann to the name of your model (without the .h5). In addition, you need to tell the toolbox where to find the model:

If you don't set the parameter path_wd in your config file, then it will be set to the folder where your config file is. In that case, you need to put your model into the folder where the config file is.

Otherwise, just set the path_wd to where your model is.

[wangcm12]

I did what you said. ,the parameter path_wd was set, the model in the folder where the config file is. but it reported " No model found in config file ”，I spent a day finding the problem, but still not solved. Help!

[rbodo]

Please send me the full error trace, your config file, and a picture of your file structure (where the model and config file are).

[wangcm12]

Thank you! Please contact me if you need me.

…

------------------ 原始邮件 ------------------ 发件人: "Bodo Rueckauer"<notifications@github.com>; 发送时间: 2018年12月18日(星期二) 晚上10:59 收件人: "NeuromorphicProcessorProject/snn_toolbox"<snn_toolbox@noreply.github.com>; 抄送: "王春妹"<980241073@qq.com>; "Author"<author@noreply.github.com>; 主题: Re: [NeuromorphicProcessorProject/snn_toolbox] convert the model ofobjection (#27) Please send me the full error trace, your config file, and a picture of your file structure (where the model and config file are). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[rbodo]

Sorry, I did not see any files.

[wangcm12]

sorry，I reply by QQ E-mail, the file upload failed. I tried convert the caffemodel,it succeeded，the config file is ok. I think I probably know the reason of the error. When I ran my Python file on my computer, it still report the same error as this"NO model in the config file", so it should be my Python or keras problem.