Mean Average Precision producing incorrect values, maybe?

[dominic-simon]

Hello,

I'm trying to find the Mean Average Precision (mAP) over a set of images from the MSCOCO dataset that have been modified to cause the object detector in use (Faster-RCNN, in this case) to predict objects that don't exist. To be clear, the objects that don't exist are objects that generally have very little to no overlap with any ground truth objects.

My issue is that when I compare the original image mAP to the modified image mAP, they are the exact same, which does not seem correct. Below are what the images look like:
Ground Truths

Original Image Predictions

Modified Image Predictions

To the best of my knowledge, these two images should have two different mAPs, with the original image having a much higher mAP than the modified image since the modified image has many False Positive predictions (all the boxes to the left of the man on the surfboard). However, that is not the case, with both images receiving a mAP of 0.85.

Can someone confirm that I am correct in thinking that these two images should receive different mAPs? If I'm wrong, then perhaps an explanation of where I'm going wrong would be useful? I'm not sure what I'm misunderstanding here, so any help is appreciated.

Here's my code, in case I'm doing something wrong there:

import torch
import torchvision.transforms as transforms
import torchvision.models.detection as models
from torchvision.datasets import CocoDetection
from torchmetrics.detection.mean_ap import MeanAveragePrecision

import os
import numpy as np
from PIL import Image
from tqdm import tqdm
import matplotlib.pyplot as plt

PATCH_SIZE = 100
FOLDER_PATH = f'./SAC_PGD/{PATCH_SIZE}'
DEVICE = 'cuda'

model = models.fasterrcnn_resnet50_fpn_v2(weights='FasterRCNN_ResNet50_FPN_V2_Weights.COCO_V1').eval().to(DEVICE)

mAP = MeanAveragePrecision()

ms_coco = CocoDetection('../datasets/MSCOCO/val2017', '../datasets/MSCOCO/annotations/instances_val2017.json')

for i in tqdm(range(14, 15)):
    # Reformatting the mscoco ground truths
    b = []
    l = []
    s = []
    for j in range(len(ms_coco[i][1])):
        bboxes = ms_coco[i][1][j]['bbox']
        corrected = [bboxes[0], bboxes[1], bboxes[0]+bboxes[2], bboxes[1]+bboxes[3]]
        b.append(corrected)
        l.append(ms_coco[i][1][j]['category_id'])
        s.append(1.0)
    gt = [{'boxes':torch.from_numpy(np.array(b)).to(DEVICE),
           'labels':torch.from_numpy(np.array(l)).to(DEVICE),
           'scores':torch.from_numpy(np.array(s)).to(DEVICE)}]

   # Open the original image 
    original = (np.asarray(ms_coco[i][0]) / 255).astype(np.float32)
    row, col = original.shape[0], original.shape[1]

    to_tensor = transforms.Compose([transforms.Resize((row, col)),
                                    transforms.ToTensor()])

   # Open the modified image and a mask showing where the modification is
    image_path = os.path.join(FOLDER_PATH, f'{i}_adv.png')
    mask_path = os.path.join(FOLDER_PATH, f'{i}_mask.png')
    image = Image.open(image_path).convert('RGB')
    mask = Image.open(mask_path).convert('L')
    image = np.transpose(to_tensor(image).numpy(), (1, 2, 0))
    mask = np.stack([to_tensor(mask).numpy()] * 3, axis=3)[0]

    # I was having some issues with data. This solves that 
    image = np.where(mask == 1, original, image)

   # Reformat modified image for use in pytorch Faster-RCNN
    image = torch.from_numpy(np.transpose(image, (2, 0 ,1))).to(DEVICE)

   # Get modified and original image predictions
    with torch.no_grad():
        output = model([image])
        b_out = model([torch.from_numpy(np.transpose(original, (2, 0 ,1))).to(DEVICE)])

   # Draw the bounding boxes onto the images
    from PIL import ImageDraw
    to_pil = transforms.ToPILImage()
    original = to_pil((original * 255).astype(np.uint8))
    attacked_image = to_pil((image * 255).type(torch.uint8))
    drawer = ImageDraw.Draw(original)
    drawer_copy = ImageDraw.Draw(attacked_image)
    bboxes = output[0]['boxes']
    bbboxes = b_out[0]['boxes']
    for box in bboxes:
        drawer_copy.rectangle([box[0], box[1], box[2], box[3]], outline=(255, 0, 0), width=3)
    #for box in ms_coco[i][1]:
    for box in bbboxes:
        #box = box['bbox']
        drawer.rectangle([box[0], box[1], box[2], box[3]], outline=(0, 255, 0), width=3)
    original.save(f'./{i}_benign.png')
    attacked_image.save(f'./{i}_adv.png')

   # Update the predictions used in the mAP. 
   # I switch b_out and output to test the original and modified images mAP, respectively. 
    mAP.update(b_out, gt)

# Compute and show the final mAP.
total_mAP = mAP.compute()
print(total_mAP['map'])

[Leonnorblad]

Any updates on this issue? I have a similar problem