How do I interpret evaluation results on my custom dataset?

[Suvi-dha]

After performing training, I ran evaluation code on 500 images to which I received following results.
Can any one help me understand these results. Any help is much appreciated.

Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.061
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.075
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.059
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.094
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.099
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.099
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.099
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.149

[qmzsuxing]

@Suvi-dha have you solved your problem? I also wonder what does this mean...

[Suvi-dha]

yes, I read about it. The values as stated are average precision and average recall on IoU(intersection over union) thresholds between 0.5 and 0.95 where maximum detections over which it is calculated is 100. In other words, precision is calculated over number of positive detections (greater than IoU threshold) by total number of TPs and FPs. and then the average of precision values over each threshold is calculated to obtain the final value in the last column.
Similarly for recall.
Also, they have calculated Average Precision (AP) and AR over different areas (all, small, medium and large)

[ashnair1]

@Suvi-dha So how exactly would this result be reported? Is your mAP = 0.061? Isn't this an extremely low value? I'm facing a similar issue where my AP score for all IOU thresholds is 0.075. I don't understand how to actually report the score of my model.

[Suvi-dha]

MaskRCNN doesn't work on my dataset due to many issues, that's why the results are low. I tried training by changing hyperparameters to see the effect. So, you can also try changing them and then observe the results. May be they will improve.
I don't completely understand what do you mean by reporting the score of your model.
If you are comparing your model performance with maskRCNN then the metrics you are using for reporting your result should also be calculated from MRCNN, or the other way round.

[ashnair1]

@Suvi-dha By reporting the score I mean reporting the mAP score since that's the common evaluation metric. I tried calculating the mAP score using the compute_ap function as in the train_shapes.ipynb and that gave me a somewhat reasonable mAP score of 0.272 or 27.2. I guess I'll just go with that.

For reference, my dataset consists of satellite images

[eyildiz-ugoe]

I am having exactly the same problem. What functions do you guys use to evaluate your model? Where are these functions located? train_shapes.ipynb only has mAP, that's not enough. We need to evaluate it using the metrics given by COCO. (AP50, AP75, AP_S...etc)

[banafsh89]

@Suvi-dha Could you please tell me which part of the code gives you Average Recall. I can only calculate AP using utils.compute_ap. I don't know how to calculate Average Recall!

[Suvi-dha]

@Suvi-dha Could you please tell me which part of the code gives you Average Recall. I can only calculate AP using utils.compute_ap. I don't know how to calculate Average Recall!

you can calculate average recall from utils.compute_recall Also, compute_ap also returns iou, precision and recalls.

[Suvi-dha]

I am having exactly the same problem. What functions do you guys use to evaluate your model? Where are these functions located? train_shapes.ipynb only has mAP, that's not enough. We need to evaluate it using the metrics given by COCO. (AP50, AP75, AP_S...etc)

Search in utils.py.

[banafsh89]

Thanks, but that is only recall in each iou range! how should I find the average recall! For example if you look at compute_ap, it finds mAP as "Compute mean AP over recall range". Do you have any idea bout that?

[Suvi-dha]

Thanks, but that is only recall in each iou range! how should I find the average recall! For example if you look at compute_ap, it finds mAP as "Compute mean AP over recall range". Do you have any idea bout that?

you have to use pycocotools library if you want to calculate the results like as shown above in my first comment, on your dataset.

from pycocotools.coco import COCO
from pycocotools.cocoeval import COCOeval
from pycocotools import mask as maskUtils

check the code for

cocoeval

.

[sineagles]

Do you solve this problem? I meet the same problem same to yours. Can you tell me how correct it?

[sineagles]

Do you solve this problem? I meet the same problem same to yours. Can you tell me how correct it?

iii

[jdmelendez]

@ Suvi-dha Entonces, ¿cómo se informará exactamente este resultado? ¿Es su mAP = 0.061? ¿No es este un valor extremadamente bajo? Me enfrento a un problema similar en el que mi puntaje AP para todos los umbrales de IOU es 0.075 . No entiendo cómo informar realmente el puntaje de mi modelo.

If you're training with COCO, the mAP score is the AP. Perhaps is this what you are looking for. Reading on http://cocodataset.org/#detection-eval , in "2.Metrics" there is the next line:

"AP is averaged over all categories. Traditionally, this is called "mean average precision" (mAP). We make no distinction between AP and mAP (and likewise AR and mAR) and assume the difference is clear from context."

[WillianaLeite]

@ Suvi-dha Você poderia me dizer qual parte do código fornece a média de recall? Só posso calcular o AP usando utils.compute_ap. Não sei como calcular o recall médio!

você pode calcular o recall médio a partir de utils.compute_recall Além disso, o compute_ap também retorna iou, precision e recalls.

@Suvi-dha Could you please tell me which part of the code gives you Average Recall. I can only calculate AP using utils.compute_ap. I don't know how to calculate Average Recall!

you can calculate average recall from utils.compute_recall Also, compute_ap also returns iou, precision and recalls.

Could you please explain to me how to calculate mAR from the "utils.compute_recall" function, I understand that it returns the AR, but how should I calculate the mAR? Please help me!!

[Joanne513]

After performing training, I ran evaluation code on 500 images to which I received following results.
Can any one help me understand these results. Any help is much appreciated.

Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.061
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.075
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.059
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.094
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.099
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.099
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.099
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.149

how you got this result? i only can use the compute_ap function to calcute the AP @ IoU = 0.5, how you got so many IoUs? can you help me!

[elia86]

@ Suvi-dha Você poderia me dizer qual parte do código fornece a média de recall? Só posso calcular o AP usando utils.compute_ap. Não sei como calcular o recall médio!

você pode calcular o recall médio a partir de utils.compute_recall Além disso, o compute_ap também retorna iou, precision e recalls.

@Suvi-dha Could you please tell me which part of the code gives you Average Recall. I can only calculate AP using utils.compute_ap. I don't know how to calculate Average Recall!

you can calculate average recall from utils.compute_recall Also, compute_ap also returns iou, precision and recalls.

Could you please explain to me how to calculate mAR from the "utils.compute_recall" function, I understand that it returns the AR, but how should I calculate the mAR? Please help me!!

Hi @WillianaLeite ! Have you found a way to calculate mAR? I have the same issue here :)

[MALLI7622]

After fine-tuning Faster-RCNN on my custom data, I was getting this error
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000

@Suvi-dha @ashnair1 Can anybody help to resolve this issue...?

[keremtatlici]

@MALLI7622 Can you show us a sample of your dataset, i mean 1 sample's mask and original image

[Olamilekan002]

It seems the model is not learning at all

[MGardien]

After fine-tuning Faster-RCNN on my custom data, I was getting this error Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000

@Suvi-dha @ashnair1 Can anybody help to resolve this issue...?

I had a similar issue. You should check the mask format. I could contain greycolors, where it should contain classes.

[babanthierry94]

After fine-tuning Faster-RCNN on my custom data, I was getting this error Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000

@Suvi-dha @ashnair1 Can anybody help to resolve this issue...?

I have the same issue. Please did someone solve it ?

[Ann-ad]

@babanthierry94 @MGardien What is your number of training steps?

[Ann-ad]

@babanthierry94 @MGardien , Try increasing the number of your training steps

[nimakasipour]

@babanthierry94 the metric works if you perform the object detection on all images in the coco dataset, i found out when i saved the ground truth bounding boxes from "instances_val2017.json" and tried to evaluate them with with these metrics (except to get 1.0 for all metrics). if some images are missing, then the result is lower or not 1.0, i used the fifty-one metrics which is also proposed on the homepage of the coco dataset, with fiftyone it was possible to evaluate the results even if they are not performed on all images

[andycga]

Can anybody help to resolve this issue...?
Accumulating evaluation results...
DONE (t=0.23s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.988
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.991
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.991
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.988
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.993
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.993
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.993
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.993

what does the -1 mean?