-
Notifications
You must be signed in to change notification settings - Fork 2
/
README
executable file
·106 lines (80 loc) · 4.5 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
Derek Hoiem
July 30, 2012 (updated July 31, 2014)
This project contains the source code and annotations for analyzing object
detectors with the PASCAL VOC 2007 dataset. The annotations were created
by Yodsawalai Chodpathumwan in Spring 2010 and Fall 2011 as an undergraduate
RA. The associated published paper is:
D. Hoiem, Y. Chodpathumwan, and Q. Dai,
"Diagnosing Error in Object Detectors", ECCV 2012.
************* How to Run ***************
CASE I: PASCAL VOC 2007, felzenszwalb or vedaldi detectors
1) In detectionAnalysisScript, set all flags on top to 1 (true)
(note: DO_SHOW_SURPRISING_MISSES is optional)
2) Set the imdir path to a valid directory of VOC images
3) Run detectionAnalysisScript in Matlab
CASE II: PASCAL VOC 2007, your own detectors
1) Create a text file of detector outfits with rows of
file_id conf x1 y1 x2 y2 (standard format, see readDetections.m).
Put this file in a subdirectory within detections.
2) Add a corresponding entry to setDetectorInfo.m and update detector variable in
detectionAnalysisScript.
3) Perform all steps of CASE I.
CASE III: later versions of VOC
Annotations are not currently available for detailed analysis of true detections.
1) Modify setDatasetParameters.m:
a) Set the imdir, VOCsourcepath, and VOCset to correspond to the dataset
b) Set objnames_extra = {}; change objnames_extra in DISPLAY_TP to
objnames_selected.
2) Run as in CASE I or II
Case IV: other datasets
If detections are bounding boxes and evaluation criteria is similar, use 'voc_compatible'
as detector type and update setDatasetParameters.m. Otherwise, more modification may be
required; see details in Code_Explanation.txt.
CREATING A REPORT
1) The tex file for a report will be created for you in results/detname/tex.
2) Automatically remove whitespace from the figures using an Adobe Acrobat
batch process or this script for pdfcrop from Ross Girshick:
#!/bin/sh
find . -name "*.pdf" -print0 | xargs -P6 -0 -I file pdfcrop file file
3) Use a latex compiler on detectionAnalysisAutoReportTemplate to make the pdf.
************* Description of Folders ***************
annotations:
Contains annotations for objects (excluding "difficult" objects, as defined
by VOC annotations). The file Labels.txt helps to explain the format.
Each file also contains whether the object was detected by the Felzenszwalb
or the Vedaldi detector and the corresponding confidence, but this need not
be used.
src:
Contains code for reading the annotations and storing them in the PASCAL VOC
record structure. Also contains code for computing normalized average
precision and making comparisons between different subsets of the data.
results:
For each detector, contains tables of normalized AP for various subsets of
data and comparison plots.
detections:
For each detector, contains a list of all detections above some threshold
for each object.
************* Description of Script/Functions ***************
detectionAnalysisScript: the main script for analysis
TP_ANALYSIS: correlates detections with ground truth, assigns attributes
such as occlusion and aspect to each object, and computes
performance measures for various subsets of objects
FP_ANALYSIS: computes fraction of top false positives due to localization
error, confusion with similar objects, etc.; also computes
the AP impact of false positives
TP_DISPLAY: creates plots showing performance of detector for different
subsets of objects, a summary plot, a text summary of
characteristics of missed objects, and images of objects
that are less confidently detected than expected
FP_DISPLAY: displays statistics of the frequency and impact of false
positives, including creation of a table in a text file
DO_TEX: creates tex files for compiling a report
Other useful functions
displayTopFP: displays most confident false positives
displayRankedPositives: shows detection confidences of a subset of objects,
from most to least confident
************* Further Help ***************
Contact dhoiem@illinois.edu if there is a problem with the annotations or code.
If instructions are unclear or incomplete, please try to figure it out first and
send me a corrected version of the instructions. If you can't figure it out,
email me, and I will help as time allows.