NVIDIA Caffe 0.17 Poor Results vs Caffe SSD

[nViso]

Results between Caffe SSD and NVIDIA Caffe 0.17 with SSD integrated do not produce the same results.

[ghimiredhikura]

I performed exactly the same procedure to reproduce results of SFD in NVIDIA version of caffe. But results are not same. I got as follows.

[ghimiredhikura]

I was trying to test performance in this version of caffe click
but having trouble to compile it click

[ghimiredhikura]

I can successfully install caffe-jacinto version of caffe, but runtest failed. although I tried to test SFD and got following error. no luck with surfing google. any idea?

python2.7 test.py -d FDDB -p ../../../datasets/FDDB/
I0404 22:53:15.328485  9908 gpu_memory.cpp:159] GPUMemory::Manager initialized with Caching (CUB) GPU Allocator
I0404 22:53:15.328891  9908 gpu_memory.cpp:161] Total memory: 3157655552, Free: 2530803712, dev_info[0]: total=3157655552 free=2530803712
I0404 22:53:15.329195  9908 gpu_memory.cpp:159] GPUMemory::Manager initialized with Caching (CUB) GPU Allocator
I0404 22:53:15.329479  9908 gpu_memory.cpp:161] Total memory: 3157655552, Free: 2530803712, dev_info[0]: total=3157655552 free=2530803712
Traceback (most recent call last):
  File "test.py", line 67, in <module>
    process_imgs_list(imgs_list, dets_file, dataset_path, dataset_name, device)
  File "test.py", line 12, in process_imgs_list
    net = SFD_NET(device=device)
  File "/home/anudee/Desktop/CAFFE_SFD/caffe-jacinto/SFD/sfd_test_code/sfd_detector.py", line 44, in __init__
    caffe.Net.__init__(self, model_file, caffe.TEST, weights=pretrained_file)
Boost.Python.ArgumentError: Python argument types in
    Net.__init__(SFD_NET, str, int)
did not match C++ signature:
    __init__(boost::python::api::object, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int)
    __init__(boost::python::api::object, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int)

[ezetl]

Hi @ghimiredhikura, thats strange, I will try both versions later today and tell you if I can find what the issue is.

[ghimiredhikura]

Thank you @ezetl ^^.

[ezetl]

Regarding Caffe 0.17, I got the same results as @ghimiredhikura (using the original model). Moreover, Caffe 0.17 seems to be about 35% slower than SSD in GPU:

(Time in secods for each test, avg over 3 runs)

version	AFW	FDDB	PASCAL	avg forward time
SSD	21.56	69.76	35.49	0.17
NVIDIA 0.17	26.66	111.04	53.64	0.23

The precision by default is FLOAT and I have a GTX 1080.

As an example, there are 45 images in AFW where the number of detected faces is different from SSD to NVIDIA Caffe.

There seems to be some difference in the implementations, I can keep digging on that.

[bonseyes-admin]

Can we add to the timing table and accuracy table also the TIDSP Jacinto version of Caffe that also integrates SSD to see if the differences in speed and accuracy are consistent. Then yes we need to work out what are the differences.

[ghimiredhikura]

@bonseyes-admin, but the problem is we can't run our script using Jacinto version of Caffe, it gives error even compile is successful using cmake or make. The error that I got is posted above. It seems error related to boost library, I even update boost library to version 1.66, but no any luck yet.

[ezetl]

Update on Nvidia Caffe 0.17: There is a strange bug when forwarding more than one image that leads to the output Blobs to be different than the SSD implementation. This happens after forwarding several images and I am not sure where the problem may be. Note that if the network gets reloaded for every image, we can reach the baseline accuracy and the model works OK, but of course that is extremely inefficient. Maybe this can be solved by using a model fully trained inside Caffe 0.17. Anyway, I opened an issue to see if someone else faced a similar problem.

Regarding Caffe-jacinto, same issues as @ghimiredhikura, will resume tomorrow.

[ezetl]

There is a PR with the fix to make the scripts work with caffe-jacinto. Could you try it @ghimiredhikura and tell me if the problem persists?.

Regarding Nvidia Caffe 0.17, I'll leave this issue open until there is a fix for #494

[ghimiredhikura]

Yes, after this small correction in net initialization and properly linking caffe python path, it just works fine. And yes I also got exactly same result as @ezetl in caffe-jacinto. Regarding Nvidia Caffe 0.17 still it gives poor results.

[ezetl]

The author of Nvidia Caffe solved the issue. There will not be a new release yet, but I tested the development branch (caffe-0.17-494) and it works. Results are now the same as in the other versions of Caffe.