How to evaluate on BSDS500 benchmark?

[zchrissirhcz]

First I would appreciate the released repos of RCF and related ploting codes.

I've trained the RCF edge detector on BSDS500 dataset (yes, originally, instead of my own dataset, and with default training hyper-parameters). Now evaluating on BSDS500 benchmark, have some problems.

As metioned in readme.md, non-maximum suppression and edge thinning, are two acquired steps before evaluation. I installed Piotr Dollar's toolbox and edge detection code, and successfully get the results of nms-ed edges, saved in png images.

However, whether doing edge thinning or not, the BSDS500 benchmark's code, always give complains that the indexes are 0, which should be greater than or equal to 1. I notice that this is caused by, the automatically generated thresholding values are from 0.01 to higher ones, and the pixel values after binarization (via bwlabel in matlab) contains 0 values.

Wondering if you have encounterd same problem, if so, would you provide some suggestions? Thank you.

[yun-liu]

@zchrissirhcz I guess that you have used the wrong wrappers. You should use the function of boundaryBench in the BSDS500 benchmark to evaluate the produced edges. Moreover, the edge thinning has been performed in the function of evaluation_bdry_image using bwmorph function in matlab.

[zchrissirhcz]

Thanks for your reply. Now I can get correct evaluation result. I'll put my steps here.

1.use RCF's caffemodel to genereate png files

Now I use the downloaded rcf_pretrained_bsds.caffemodel to generated multi-scaled results, saved as *_fuse.png.

2. do nms

I use your provides edge_nms.m script, and find it cannot use edgesNms funtion, thus I move it to Piotr Dollar's edges folder, i.e. /opt/work/piotr/edges, and change this line:

edge = edgesNms(edge, O, 2, 5, 1.01, 8);

to

edge = edgesNmsMex(edge, O, 2, 5, 1.01, 8);

Then run it to generate nms-ed results.

Note, I've also downloaded Piotr Dollar's toolbox and both these two folders(and children folders) are added to path.

3. use BSDS benchmark scripts for boundary evaluation

I downloaded http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/BSR/BSR_full.tgz, and by editing bench_bsds500.m to these lines:

addpath benchmarks

clear all;close all;clc;

imgDir = '../BSDS500/data/images/test';
gtDir = '../BSDS500/data/groundTruth/test';
%inDir = '../BSDS500/ucm2/test';
%outDir = '../BSDS500/ucm2/test_eval';

inDir = '/opt/data/HED-BSDS/test-fcn-nms';
outDir = '/opt/data/HED-BSDS/RCF-eval';

%mkdir(outDir);

% running all the benchmarks can take several hours.
tic;
%allBench(imgDir, gtDir, inDir, outDir)
boundaryBench(imgDir, gtDir, inDir, outDir);
toc;

plot_eval(outDir);

Then I run that script, and after a long time(several hours) waiting, some .txt files are generated, including eval_bdry.txt and eval_bdry_thr.txt.

I thought eval_bdry.txt and eval_bdry_thr.txt can be put under your provided plotting code folder, and I run https://github.com/yun-liu/plot-edge-pr-curves/blob/master/plot_bsds.m.

However, errors just occurs. Notice that there are exactly 7 numbers (7 columns) in *_bdry.txt in plot-edge-pr-curves/edge_eval_bsds, and in BSR/bench/benchmarks/collect_eval_bdry.m, there is:

    fname = fullfile(pbDir,'eval_bdry.txt');
    fid = fopen(fname,'w');
    if fid==-1,
        error('Could not open file %s for writing.',fname);
    end
    fprintf(fid,'%10g %10g %10g %10g %10g %10g %10g %10g\n',bestT,bestR,bestP,bestF,R_max,P_max,F_max,Area_PR);
    fclose(fid);

i.e. The BSDS boundary benchmark will generate eval_bdry.txt with exactly 8 numbers, instead of 7.
Similarly, my generated eval_bdry_thrs.txt have 4 columns, and your provides *_bdry_thr.txt have 3 columns.

4. removing the first columns in eval_bdry.txt and eval_bdry_thr.txt

This is claimed in https://github.com/yun-liu/plot-edge-pr-curves/blob/master/README.md

5. do plot

modify and run plot-edge-pr-curves/plot_bdsd.m

Now I get a nearly exactly same result:

ODS=0.806 OIS=0.823 AP=0.839 R50=0.914 - RCF (2017)
ODS=0.811 OIS=0.830 AP=0.846 R50=0.913 - RCF-MS (2017)
ODS=0.811 OIS=0.831 AP=0.847 R50=0.916 - MyRCF (2018)

MyRCF_bdry.txt:

   0.796388   0.826376   0.811105   0.823153   0.838022   0.830521   0.846884

MyRCF_bdry_thr.txt:

0.916885	0.481511	0.631423
0.913225	0.533613	0.673619
0.911576	0.552105	0.687699
0.909399	0.571562	0.701947
0.908081	0.581843	0.709245
0.906144	0.59488	0.718239
0.904932	0.602573	0.72343
0.903104	0.612941	0.730254
0.901829	0.619326	0.734345
0.89963	0.628464	0.739987
0.897387	0.637358	0.745345
0.89596	0.642711	0.748495
0.893796	0.6503	0.752849
0.89235	0.654891	0.755399
0.890059	0.661843	0.759171
0.888551	0.666303	0.761543
0.886357	0.672549	0.764791
0.884839	0.676687	0.76689
0.882561	0.682873	0.769981
0.881107	0.686943	0.772004
0.878687	0.693304	0.775065
0.876151	0.699512	0.777931
0.874373	0.703607	0.77975
0.871617	0.709732	0.782388
0.869685	0.713748	0.784038
0.866786	0.719888	0.786536
0.864841	0.723922	0.788132
0.8619	0.729702	0.790311
0.859778	0.73352	0.791646
0.856537	0.739282	0.793602
0.853173	0.744997	0.795424
0.851031	0.74876	0.796626
0.847647	0.754523	0.798379
0.845409	0.758379	0.799533
0.842058	0.763776	0.801009
0.839689	0.767495	0.801971
0.836156	0.773108	0.803397
0.833805	0.776892	0.804343
0.830269	0.782603	0.805732
0.827745	0.786396	0.806541
0.823914	0.792099	0.807694
0.819892	0.797895	0.808744
0.817159	0.801629	0.80932
0.812873	0.807348	0.810101
0.809865	0.811157	0.81051
0.80516	0.816768	0.810922
0.801654	0.820633	0.811032
0.796388	0.826376	0.811105
0.792854	0.830137	0.811067
0.787342	0.835859	0.810875
0.781505	0.841581	0.810432
0.777526	0.845342	0.810017
0.771536	0.850996	0.809321
0.767256	0.854771	0.808653
0.760605	0.860382	0.807422
0.755853	0.864141	0.806378
0.748499	0.86977	0.80459
0.743674	0.873558	0.8034
0.736299	0.879468	0.801541
0.731124	0.883238	0.800014
0.722958	0.888869	0.797375
0.714331	0.894332	0.794261
0.70829	0.897902	0.791904
0.698607	0.902929	0.787735
0.692028	0.906222	0.784772
0.681991	0.911481	0.78021
0.674913	0.914805	0.776759
0.66391	0.91985	0.7712
0.656116	0.923474	0.767169
0.643958	0.928576	0.76051
0.630972	0.933331	0.75293
0.621734	0.936501	0.747326
0.607383	0.941441	0.738386
0.597245	0.944526	0.731773
0.581938	0.948922	0.721443
0.570807	0.951668	0.713599
0.553205	0.955427	0.700697
0.540565	0.958061	0.691159
0.51997	0.961771	0.675006
0.505487	0.964148	0.663245
0.482445	0.967519	0.643843
0.458149	0.970794	0.622514
0.441003	0.972765	0.606878
0.413966	0.976042	0.581361
0.394876	0.977813	0.562568
0.364598	0.981287	0.531658
0.34286	0.983228	0.508428
0.30815	0.986119	0.469567
0.283816	0.988241	0.440984
0.244985	0.991295	0.392876
0.20395	0.993577	0.338431
0.176363	0.994982	0.299618
0.134409	0.996868	0.236879
0.106832	0.998216	0.193007
0.0671202	0.999695	0.125794
0.0440549	0.999961	0.0843918
0.0173671	1	0.0341413
0.00601356	1	0.0119552
0.000150339	1	0.000300633