computeSaliency does not get correct results

[fortuneko]

i use BING as algorithm type to compute the boundingbox of an image, the sample project "computeSaliency" runs fine ,but the result is not ok, it gets too many boxes.

i painted the results on the source image ,like this

i compile opencv contrib with opencv 3.0, i wonder to know has anyone successfully run the example and get right results? what the problem may i meet?

[fpuja]

Hi @justinkoo , I am the author of the saliency module. BING algorithm works well, I checked with several tests and was also used by another research group. I think you have misunderstood its functioning: BING, instead of proposing all windows possible that you would have if you apply classical sliding window object detection paradigm, produces a small set of candidate object windows. Among them, there is certainly the target of interest. This subset, can be given as input to other algorithms of detection etc, making the job much simpler and less computationally expensive. In the example that I have inserted in the module, I have highlighted only the rectangles corresponding to the target, to highlight that among the candidates proposed by BING there are always those that you were looking for. Generally, although the proposals vary between 2000 and 5000 candidates, you can find the target of interest in the first 100-200 positions.

[fortuneko]

Hi @fpuja ,Sorry for replying so late. So as you said that the code has been tested and can get the right results, then the problem may come from the way i use it.
I use the sample project "(sample)computeSaliency" for testing , then use input paras like this "BING D:\Videos\cam1.avi 1 E:\opencv3.0\Contrib\modules\saliency\samples\ObjectnessTrainedModel"
in the ObjectnessTrainedModel fold i unpack the .gz package to yml file，like this:
E:\opencv3.0\Contrib\modules\saliency\samples\ObjectnessTrainedModel
\ObjNessB2W8HSV.idx.yml
\ObjNessB2W8HSV.wS1.yml
......
the yml files are in the same ObjectnessTrainedModel fold .

anything wrong with the way i using this code?

[fpuja]

Hi @justinkoo , sorry you for replying so late. Well, looking at your procedure, the pipeline to run the code it's ok, but you must NOT unpack the .gz, you must collocate in the folder the .gz file directly. This format is designed to save space, and the code is already enabled to make the reading in this format.

So now you can try again the code, but N.B. as I have wrote in the previous answer: the algorithm give as output 2k-5k bounding box, among which there are your target of interest.

[fortuneko]

Hi@fpuja,I have spent some time to read the paper of this code and test its accompanying code ,the result is similar as the module packaged by you.
i have a misunderstanding about the result,the experience data the paper talking about is recall rate, not the average precision rate, though its recall rate maybe high ,but it results too many proposals, i need to post select these proposals again by myself,that's still need a lot of time .Again ,the proposed box do not extractly locate the real object ,someone here do a experience about the objectness algorithm including BING.https://pdollar.wordpress.com/2014/11/18/evaluating-object-proposals/

[fpuja]

Hi @justinkoo , I also performed tests using my code and their and the results are the same :)
As for the amount of the results, as I said, there is a wrong understanding that I have had too the first time I read the paper. BING does not provide the only bounding box-containing the object; the algorithm provides a set of boxes (among which for sure there is the searched object) to feed to additional algorithms. The sense is: most of the algorithms of object detection and recognition, rely on a sliding window processing, much computationally heavier and returning a much bigger guess. BING, which is very fast due to its theory and implementation, ensures a very rapid preprocessing phase, providing the reliable guess on which to perform further processes of detection and recognition.

[fpuja]

anyway @justinkoo, if you have further doubts about the theory of the algorithm, perhaps should write directly to the author, with whom I had an email exchange and is very friendly and helpful. :) If then you will also contribute to my module maybe adding another objectness or object detection algorithm, I would be really glad! :)

[fortuneko]

thanks @fpuja ,if i have time ,i will have a try^-^

[antran89]

Hi @fpuja,
I am testing BING in OpenCV with this simple code. I discover that the speed of BING is really worse than reporting in the original paper. It takes around 3 seconds for an image. What is the possible reason? If it is slow, than BING will its interests in other approaches.

#include <opencv2/saliency.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/imgproc.hpp>

#include <iostream>
#include <chrono>
#include <ctime>

using namespace cv;
using namespace std;
using namespace cv::saliency;

const String keys =
    "{help h usage ? |        | print this message   }"
    "{@imagepath     |        | image1 for compare   }"
    ;

int main(int argc, char** argv)
{
    cout << "OpenCV version: " << CV_VERSION << endl;

    CommandLineParser parser(argc, argv, keys);
    string imagePath = parser.get<string>(0);
    Mat img = imread(imagePath);

    // initialize BING algorithm
    ObjectnessBING objProposal;
    string training_path = "/home/tranlaman/Downloads/opencv-new/opencv_contrib/modules/saliency/samples/ObjectnessTrainedModel";

    vector<Vec4i> saliencyMap;
    objProposal.setTrainingPath(training_path);

    // display some information about BING

    cout << "getBase() " << objProposal.getBase() << endl;
    cout << "getNSS() " << objProposal.getNSS() << endl;
    cout << "getW() " << objProposal.getW() << endl;

    // timing
    using namespace std::chrono;
    steady_clock::time_point t1 = steady_clock::now();

    // do computation.
    objProposal.computeSaliency(img, saliencyMap);

    steady_clock::time_point t2 = steady_clock::now();
    duration<double> time_span = duration_cast<duration<double>>(t2 - t1);
    cout << "Running time: " << time_span.count() << " seconds!" << endl;

//        vector<float> objectnessScores = saliencyAlgorithm.dynamicCast<ObjectnessBING>()->getobjectnessValues();

    for (int i = 0; i < std::min<int>(saliencyMap.size(), 50); i++)
    {
        Mat image = img.clone();
        rectangle(image, Point(saliencyMap[i][0], saliencyMap[i][1]), Point(saliencyMap[i][2], saliencyMap[i][3]), Scalar(255, 0, 0));
        imshow("object proposals", image);
        waitKey();
    }

}

And the running time for an image:

Average time for predicting an image (MAXBGR) is 0.195671s
Average time for predicting an image (HSV) is 0.208561s
Average time for predicting an image (I) is 0.188055s
Running time: 3.28662 seconds!```

[ramanpreet9]

@antran89 i tried running your code but its giving me errors:
/home/raman/Work/obj_prop_3D/main.cpp:168: error: undefined reference to `cv::saliency::ObjectnessBING::ObjectnessBING()'

/home/raman/Work/obj_prop_3D/main.cpp:172: error: undefined reference to `cv::saliency::ObjectnessBING::setTrainingPath(std::string)'

/home/raman/Work/obj_prop_3D/main.cpp:185: error: undefined reference to `cv::saliency::Saliency::computeSaliency(cv::_InputArray const&, cv::_OutputArray const&)'

etc.

i changed the model training path correctly. ANy idea why this is happening?

[fpuja]

Hi @ramanpreet9,

I think this closed issue is not the correct place to write, because you have not problem with algorithm results, you have a building errors and, as you know, is very different :)

Your building problems are not due to wrong model training path, you have problem with open cv library and my module. If you are on UNIX sistem, check that you are using or you have linked the correct open cv and open cv contrib correct folder

[ramanpreet9]

Sorry about posting here.. i didnt realize the thread was closed.

I double checked my links. I think the problem is that your implementation in opecv_contrib has been edited a bit which isnt letting us compile the program correctly. Another frien dof mine is having same issue with ObjectnessBING as well.

Will it be possible for you to provide a sample code that takes an image and runs bing on it returning say 1000 bounding boxes along with its value on the opencv 3.0 (with bing in opencv_contrib part) ?

[fpuja]

Sorry but ,trust me , you are thinking wrong :)

My module on opencv_contrib has no problem, otherwise it would be not on opencv_contrib ;) I you are not a neophyte on OpenCV stuff and community, you know that a module, or proposed change on a module, are merged in a opencv or opencv_contrib only after passing different steps, the first of which is to pass a cross platform building.

So again, trust me, the problem is on your machine :) You have problems with opencv configuration - linking - including. Resolve this problem and all works well ;)

best

[ramanpreet9]

ok, perhaps you are right =). For my cmake file this is what i have:
`project(obj_prop_3D)
cmake_minimum_required(VERSION 2.8)
aux_source_directory(. SRC_LIST)

find_package(PCL REQUIRED COMPONENTS common io)
include_directories(${PCL_INCLUDE_DIRS} )
link_directories(${PCL_LIBRARY_DIRS})
add_definitions(${PCL_DEFINITIONS})

include_directories({/home/raman/Work/obj_prop_3D})
set(project_SOURCES
/home/raman/Work/obj_prop_3D/main.cpp
/home/raman/Work/obj_prop_3D/getfilenames.cpp
/home/raman/Work/obj_prop_3D/getheatmap.cpp
)
set(project_HEADERS
/home/raman/Work/obj_prop_3D/getfilenames.h
/home/raman/Work/obj_prop_3D/getheatmap.h
)
#set(OpenCV_INCLUDE_DIRS "${OpenCV_INSTALL_PATH}/include/opencv;${OpenCV_INSTALL_PATH}/include")

find_package(OpenCV 3.1.0 REQUIRED COMPONENTS core highgui imgproc imgcodecs)
#include_directories(-L/usr/local/lib -lopencv_core -lopencv_highgui -lopencv_imgcodecs -lopencv_saliency -lboost_system)
set( CMAKE_CXX_FLAGS "-g -Wall")

add_executable(${PROJECT_NAME} ${project_SOURCES})

target_link_libraries(obj_prop_3D ${PCL_LIBRARIES} ${OpenCV_LIBS} )

`

should i be adding something extra?

[fpuja]

Reflect on the commented line in your cmake. Why opencv include and install path are commented? E why the include_directories?

[ramanpreet9]

I am entirely new to opencv, cmake files etc. So most of what i have done is through SO, github threads and some ROS related stuff. i uncommented that line and edded -lopencv_contrib but it didnt work either.

Regarding the commented lines earlier: I had commented them out to put it in a pcl related thread as the issue was related to pcl. I forgot to uncomment them to link here. =) My mistake. they are all uncommented now and the updated cmake is here:

"project(obj_prop_3D)
cmake_minimum_required(VERSION 2.8)
aux_source_directory(. SRC_LIST)

find_package(PCL REQUIRED COMPONENTS common io)
include_directories(${PCL_INCLUDE_DIRS} )
link_directories(${PCL_LIBRARY_DIRS})
add_definitions(${PCL_DEFINITIONS})

include_directories({/home/raman/Work/obj_prop_3D})
set(project_SOURCES
/home/raman/Work/obj_prop_3D/main.cpp
/home/raman/Work/obj_prop_3D/getfilenames.cpp
/home/raman/Work/obj_prop_3D/getheatmap.cpp
)
set(project_HEADERS
/home/raman/Work/obj_prop_3D/getfilenames.h
/home/raman/Work/obj_prop_3D/getheatmap.h
)
set(OpenCV_INCLUDE_DIRS "${OpenCV_INSTALL_PATH}/include/opencv;${OpenCV_INSTALL_PATH}/include")

find_package(OpenCV 3.1.0 REQUIRED COMPONENTS core highgui imgproc imgcodecs )
include_directories(-L/usr/local/lib -lopencv_core -lopencv_highgui -lopencv_imgcodecs -lopencv_saliency -lboost_system -lopencv_contrib)
set( CMAKE_CXX_FLAGS "-g -Wall")

add_executable(${PROJECT_NAME} ${project_SOURCES})

target_link_libraries(obj_prop_3D ${PCL_LIBRARIES} ${OpenCV_LIBS} )

"

[ramanpreet9]

pkg-config --libs opencv

returned: -L/usr/local/lib -lopencv_shape -lopencv_stitching -lopencv_objdetect -lopencv_superres -lopencv_videostab -lopencv_calib3d -lopencv_features2d -lopencv_highgui -lopencv_videoio -lopencv_imgcodecs -lopencv_video -lopencv_photo -lopencv_ml -lopencv_imgproc -lopencv_flann -lopencv_viz -lopencv_core

so perhaps i need to add contrib manually?

[fpuja]

Yes you must do different thinks. Sorry but I can't give you a step by step guide on a openCV installation /configuration, I must work ahah :) This issues are only related to problem with my module. For the opencv configuration problem, you can find many tutorials and posts on line ;) Good luck,

best

[fpuja]

Hi @antran89,

I don't remember the paper's declared speed, it's been too long since time since I port/implement the algorithm, bu two points:

1. If you want to compare with speed reporting in the experiments section of the original paper, you must run the code on the SAME hardware and configuration, you know that right? Or you must find a way to normalize the results of your run and the paper run, otherwise you cannot compare :)

2)I'm not the author or coauthor of paper, so this is my porting/implementation of algorithm for OpenCV; I've never declare that the current implementation have the same speed as the paper declared speed.

Best