Wrong category indexing in coco evaluation (might be the cause for AP(novel)>AP(base))

[Jonas-Meier]

I probably found an issue with the indexing of categories at coco evaluation. This might cause interchanged results for each category which would also affect AP(novel) and AP(base) since it is unknown which result belongs to which category #30 . The overall mAP is unaffected by this issue.

At inference, the test-class puts the detections inside the variable all_boxes where the classes are indexed by their position in imdb.classes (which is Background-Class + Base-Classes + Novel-Classes, as defined in the constructor of coco.py).

As coco.py does the evaluation, the following happens:

• its method evaluate_detections gets the detections via all_boxes from the test class
• evaluate_detections calls _write_coco_results_file
• _write_coco_results_file iterates over its categories (self.classes) and uses the mapping coco_cat_id = self._class_to_coco_cat_id[cls] to obtain the category id from category name. It passes each category ID to _coco_results_one_category which will return detections in a different format:
  • x,y,w,h instead of xmin, ymin, xmax, ymax
  • image id instead of image index
  • category ID instead of category
• Now we have saved the detections in a different format to a json file which will be passed to _do_detection_eval
• _do_detection_eval creates an instance of COCOeval with
  • itself (the COCO object, initialized with the validation-annotation file)
  • another COCO-object initialized with the previously created json-file (the rewritten detections)
• _do_detection_eval runs evaluate and accumulate on the cocoeval object and passes it to _print_detection_eval_metrics
• inside COCOeval this takes place:
  • in its constructor, it sets self.params.catIds = sorted(cocoGt.getCatIds()), where cocoGt is the COCO-object initialized with the validation annotation file
  • evaluate() uses those catIDs to identify categories
  • accumulate() stores precision and recall of a category at the index of that category in catIDs (stores them in self.eval)

Now we have two problematic situations inside _print_detection_eval_metrics() method of coco.py:

• printing of class-wise AP:
  • directly accesses cocoeval.eval['precision'] with class incides from cls_ind, cls in enumerate(self.classes), but as stated above, the metrics for a class are stored at the index of that class as in the validation annotation file. This causes the category names for per-category results to be interchanged
• printing of summarized novel class mAP and base class mAP:
  • passes range(0, len(base_classes)) for summary of base classes and range(len(base_classes), len(all_classes)) for summary of novel classes to cocoeval.summarize. However, the summarize method uses the categoryId argument to directly access the precision and recall of that class, but those indices are wrong (as described above for class-wise AP)

To solve the stated problems, I would suggest the following changes (for _print_detection_eval_metrics, line 245-259)

cat_ids = self._COCO.getCatIds()
cats = self._COCO.loadCats(cat_ids)
cat_name_to_ind = dict(list(zip([c['name'] for c in cats], range(len(cats)))))
for cls_ind, cls in enumerate(cat_name_to_ind.keys()):
    # no check for cls == '__background__' needed due to new list we're iterating over
    precision = coco_eval.eval['precision'][ind_lo:(ind_hi + 1), :, cls_ind, 0, 2]  # no index shift necessary
    ap = np.mean(precision[precision > -1])
    print('{}: {:.1f}'.format(cls, 100 * ap))

print('~~~~ Summary Base metrics ~~~~')
categoryId = list(map(lambda cls: cat_name_to_ind[cls], self._base_classes))  # use correct indices now
coco_eval.summarize(categoryId)

print('~~~~ Summary Novel metrics ~~~~')
categoryId = list(map(lambda cls: cat_name_to_ind[cls], self._novel_classes))  # use correct indices now
coco_eval.summarize(categoryId)

self._base_classes and self._novel_classes are the lists of base and novel class names which I used to create self._classes in the constructor.

Some final thoughts on the issue:

• I think using a variable name categoryId inside cocoeval.summarize is a bit confusing, since they treat them as indices
• The crucial mistake presumably was the different order of the categories inside self.classes (of coco.py) and the categories as in the COCO object
  • In coco.py of Meta-RCNN they leave the categories in the same order as they are read in from the COCO API and just prepend a background class. That's why they are able to directly iterate over self.classes (instead of having to read in original coco categories for the correct order) and just have to do a simple index shift to obtain correct results for each class.

[YoungXIAO13]

Hi @Jonas-Meier

Thanks for spotting it and sorry for being not very responsive at this moment. This might be an actual issue, I will look further into it. My guess is that a new training strategy needs to be found if this is actually the case.

And potentially I will update the code with newer few-shot object detection code built upon detectron such like frustratingly simple ICML-2020 or contrastive CVPR-2021.