Why the speed in minicaffe is much worse than caffe with same prototxt

[yonghenglh6]

I compare the resnet from your run_test.cpp.
But the performance is like below. The speed drop down than caffe.
Any ideas?
I'm useing the newest caffe with cudnn 5.1.5.

Thanks

[luoyetx]

would you mind to paste your test code?

[yonghenglh6]

In mini-caffe, I just add a "for(int i=0;i<100;i++)" before "test.Forward();" in run_net.cpp, and get the time from your output.
In caffe, I use the caffe time, which is a mistake for memory use because it includes the backward process.

[yonghenglh6]

I get the memory use from "nvidia-smi" with watching in the flesh.

[yonghenglh6]

would u like to put up your performance with the resnet for a bug checking of mine? Thank you.

[luoyetx]

I will test the network prototxt on 1070 later. With more details on mini-caffe and official caffe.

[yonghenglh6]

I use those code to test your every layer's time. But cannot find the reason.

And the result is:

[yonghenglh6]

Your net->Forward(2,3) give me an error. So I can only use net->Forward(0,x) to get the time from the begin and sub them.

[yonghenglh6]

As the net get longer, the performance diff between mini-caffe and caffe become larger, when I test them by adding layer step by step for net construction.

[yonghenglh6]

Update the performance up.

[yonghenglh6]

I checked the cudnn and assured it ran well by adding some output info.

[luoyetx]

please refer to profile.md to check the layer wise performance. I am writing tools to do the network benchmark.

[yonghenglh6]

I have tried Profile in the beginning, but the time shown in chrome is not consistent with my test result. However, it's most possible that I made the wrong usage way.

[luoyetx]

Pay attention to the Timer, it's not accuracy, use Profiler::Now() instead.

[luoyetx]

@yonghenglh6 You can try the benchmark branch, I modify the Profiler log, it now prints layer name not layer type. You can get same layer wise performance through profile.json

[luoyetx]

I find the performance is not stable under Windows platform, I will test on Linux later.

[yonghenglh6]

With your new benchmark tool, I found the bn layer is the main part that cause the difference, your bn is cost twice time than conv. So I changed to test it on google without bn layer and the result shows similar performance between caffe and mini-caffe.
Here is the details. By the way, my caffe use atlas lib, which I think it's not important.

[luoyetx]

There is an optimization in this commit for BatchNorm Layer. I will update this from official caffe.

[yonghenglh6]

Everytime you request memory from pool, the blob will be in uninitial state, then it will call gpu_memset function in "to_gpu()", which will cost about 10% time more than original caffe whose blob will be keeped and in head_on_gpu state.

[luoyetx]

The default behave is the same in official Caffe here.

[yonghenglh6]

Yes, but the official Caffe need not reallocate blob every forward and not call the function frequently. But your minicaffe keeping on setting the blob state to uninitial to reuse the memory and then rememset the memory when calling to_gpu(), which has caused a performance problem.
It can be optimized by designing the memset moment in layers carefully instead of doing it everytime "to_gpu()" called.
In short words, this may be a problem of only minicaffe, even the code is the same

[luoyetx]

you mean this function? It is a problem that this function called every time a new memory is requested. I think we can remove this function call, as the dirty data in the memory seems no problem for late use. What do you think?

[yonghenglh6]

Yes.